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PREFACE 


Throughout  llit‘  It-xl  of  lliis  paper,  reference  is  made  to  volumes  I  through  \  .  These  volumes  have  been 
published  as  separate  leehnieal  papers  identified  as  follows: 

Volume  I 

Baum.  D.R..  Modriek.  J.A..  &  Hollingsworth.  S.R.  Irani  training  (nr  voinnwiul  anil  control  systems: 
Stains.  AFHRL-TP-82-7.  U  righl-Patlerson  AFB.  OH:  l.ogisties  and  Teehnieal  Training  Division.  Air 
Forte  Human  Resources  Laboratory.  April  1982. 

Volume  II 

Modriek.  J  \..  Baum.  D.R..  &  Hollingsworth.  S.R.  Train  training  for  rominanil  and  control  systems: 
Hrcom  mctula  I  ions  for  research  progro  m.  AFIIRL-TP-82-8.  Uright-Pallersoii  AFB.  OH:  l.ogislie-  and 
Technical  Training  Division.  Air  Force  Human  Resources  Laboratory.  April  1982. 

Volume  III 

Baum.  D.R..  Modriek.  J.  A..  X  Hollingsworth.  S.R.  Tram  training  for  command  and  control  systems: 
Recommendations  for  application  of  current  technology.  AFHRI.-TP-82-h.  S  right-Patlerson  AFB. 
OH:  Logistics  and  Technical  Training  Division.  Air  Force  Human  Resources  Laboratory.  April  I '182. 

Volume  IV 

Hollingsworth.  S.K..  Modriek.  J.  A..  it  Baum.  D.R.  Team  training  for  command  and  control  systems: 
Recommendations  for  simulation  facility.  AFHRL-TP-82-10.  Wrighl-Palterson  AFB.  OH:  Logistics 
and  Technical  Training  Division.  Air  Force  Human  Resources  Laboratory.  April  1982. 

Volume  V 

Baum.  D.R..  Modriek.  J.  \-  &  Hollingsworth.  S.R.  Team  training  for  command  and  control  systems: 
Executive  summary.  AFHRI.-TP-82-1 1.  W  riglil-Patterson  AF’B.  OH:  Logistics  and  Technical  Training 
Division.  Air  Force  Human  Resources  Laboratory.  April  1982. 

This  paper  is  the  first  of  five  volumes  prepared  by  Honevwell  to  document  the  results  of  a  research 
program  to  evaluate  the  current  status  of  team  training  (T2)  for  operators  of  complex  Air  Force  command  and 
control  (  AFC2)  systems,  and  to  make  recommendations  for  enhancing  the  AF'('2T2  process.  The  research  was 
performed  for  the  Air  Force  Human  Resources  Laboratorv  under  contract  F336 1 

This  paper  contains  the  objectives  and  approach  of  the  studv.  the  results  of  a  survey  to  assess  the  status 
of  team  training  for  command  and  control  in  Tactical  Air  (Command,  and  a  discussion  of  issues  and  problem 
areas  which  can  be  addressed  by  developing  solutions  within  currently  available  technology  and  those  that 
require  more  long-term  research  to  develop  solutions. 

This  research  effort  supports  a  major  new  Air  Force  Human  Resources  Laboratory  research  and 
development  program  whose  primary  objective  is  to  improve  T2  technologies  in  areas  particularly  relevant  to 
Air  Force  combat  readiness.  The  program  objective  requires  the  establishment  of  a  baseline  data  base  on  how 
T*  is  currently  conducted  in  the  Air  Force:  how  it  is  developed,  implemented,  and  evaluated.  Because  Air 
Force  teams  vary  greatly  in  sire,  structure,  and  functions,  it  would  be  impractical  to  eollert  data  on  the 
training  provided  to  all  of  them.  Rather,  the  scope  of  this  research  effort  had  to  be  directed  at  an  area  with 
potential  high  payoff  for  increased  combat  readiness  and  effectiveness.  The  area  of  command  and  control 
was  chosen  as  a  point  of  departure  for  the  research  because  C2  trains  lend  to  be  well-defined  structurally,  arc 
of  a  manageable  sire,  and  perform  functions  highly  representative  of  Air  Force  mission  needs.  Furthermore, 
as  the  research  effort  unfolded,  limited  time  and  resources  made  it  necessary  to  focus  on  tactical  and  air 

defense  C.2  s\ stems  to  . . xeiusion  of  strategic  C.2  systems.  Thus,  the  I'2  systems  surveyed  are.  or  in  the  case 

of  planned  systems  will  become.  Tactical  Air  Gmtmand  (TAC.)  resources. 


i 


I  In-  goal  of  this  i-fforl  was  to  develop  a  picture.  through  interview  anil  observation.  of  lion  \|V.2T2  i- 
currently  developed.  iin|ili-mriili-<l.  and  i-\aluali'd.  and  what  I'.2  training  m-i-ds  will  arise  in  the  fuluri-.  lia-ed 
on  this  picture.  capabilities.  limitations.  and  weaknesses  of  \H.2T2  won-  idi-nlifii-d  and  recommendations 
were  developed  in  three  ari  as: 

•  T-  research  and  development  program  (Volunn-  111 

•  Ri-soluhon  of  issui-s  using  current  lech  mques/lech  oologies  (\olumi-  III) 

•  Simulation  leehnologv  di-vi-lopnu-nl  for  C-T2  (\oltimc  \  ) 

Tlti-st-  recommendations  will  form  i In  foundation  lor  fuluri-  research  In  \FHKL  into  tin-  performance 
of  I.2  Iranis  and  systems.  The  research  will  encompass  training  technology,  performance  measurement 
techniques  for  C.2  teams  and  systems,  human  resources  issues  in  the  design  and  operation  of  C.2  systems,  and 
training  ot  command/ decision  skills.  The  ultimate  goal  of  this  program  is  to  imprint-  technologies  in  areas  of 
team  and  human  factors  related  to  the  eomhat  effectiveness  of  \ir  Force  II2  operations. 
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CHAPTER  I 


INTRODUCTION 


This  report  1)  discusses  the  background,  scope,  rationale,  and  approach 

2  2 

of  the  effort;  2)  reports  the  results  of  a  survey  of  AFC  T  programs, 
o 

and  3)  identifies  T  issues  and  problem  areas  in  need  of  resolution  to 

2  0 
improve  T  technology  for  operators  of  AFC“  systems. 

This  study  supports  a  major  new  Air  Force  Human  Resources  Laboratory 

research  and  development  program  whose  primary  objective  is  to  improve 
2 

T  technologies  in  areas  particularly  relevant  to  Air  Force  combat 

readiness.  The  program  objective  requires  the  establishment  of  a 

2 

baseline  data  base  on  how  T  is  currently  conducted  in  the  Air  Force; 

how  it  is  developed,  implemented,  and  evaluated.  Because  Air  Force 

teams  vary  greatly  in  size,  structure,  and  functions,  it  would  be 

impractical  to  collect  data  on  the  training  provided  to  all  of  them.  Rather, 

the  scope  of  this  research  effort  had  to  be  directed  at  an  area  with  potential 

high  payoff  for  increased  combat  readiness  and  effectiveness.  The  area 

of  command  and  control  was  chosen  as  a  point  of  departure  for  the  research 
2 

because  C  teams  tend  to  be  well-defined  structurally,  are  of  a  manageable 

size,  and  perform  functions  highly  representative  of  Air  Force  mission 

needs.  Furthermore,  as  the  research  effort  unfolded,  limited  time  and 

2 

resources  made  it  necessary  to  focus  on  tactical  and  air  defense  C 

2  2 

systems  to  the  exclusion  of  strategic  C  systems.  Thus,  the  C  systems 
surveyed  are,  or  in  the  case  of  planned  systems  will  become.  Tactical 
Air  Command  (TAC)  resources. 


1 


PROGRAM  OBJECTIVE  AND  RESEARCH  SCOPE 


% 

I 


1 

« 

The  recommendations  made  as  a  result  of  this  effort  will  form  the  foun- 

2  2 

dation  for  future  AFHBL  research  in  AFC  T  including  the  performance 

2 

of  both  teams  and  systems  in  C  .  The  ultimate  goal  of  the  program  is  to 
improve  technologies  in  areas  of  team  and  human  factors  related  to  the 
combat  effectiveness  of  Air  Force  ground  operations.  The  scope  will 

encompass  training,  performance  measures  and  techniques  for  systems 

2  * 

and  teams,  human  resources  in  C  design  and  operation,  and  training  of 

command  /  decision  s  kills . 


This  study  was  conducted  with  the  goal  of  developing  a  picture  through 

2 

interview  and  observation  of  how  T  is  currently  developed,  implemented, 

2 

and  evaluated  in  the  Air  Force,  and  what  C  training  needs  will  arise  in 

the  future.  Based  on  this  picture,  capabilities  and  limitations  of  current 
2 

Air  Force  T  were  identified.  Issues  and  problem  areas  that  need 

2  2 

resolution  to  improve  AFC  T  are  summarized  and  recommendations 
were  developed  in  three  areas  : 

2 

•  T  research  and  development  program  to  address  problems 
not  within  the  current  state  of  the  art 

•  Resolution  of  issues  using  current  techniques /technologies 

2  2 

•  Simulation  technology  development  for  C  T 
These  areas  are  expanded  in  the  other  reports. 
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RESEARCH  RATIONALE  AND  APPROACH 


In  order  to  conduct  a  meaningful  research  effort,  it  was  necessary  at  the 

2 

outset  to  restrict  the  number  of  C  systems  to  be  surveyed.  The  area  of 
2 

C  in  the  Air  Force  is  extremely  complex  and  large.  The  rationale  and 
events  that  impacted  the  research  focus  are  discussed  in  this  section. 

Some  of  the  problems  which  one  must  face  in  performing  an  exhaustive  or 

2  2 

even  comprehensive  survey  of  tactical  air  C  T  can  be  illustrated  by 
considering  the  network  depicted  in  Figure  1.  (see  Table  1  for  a  key  to 
acronyms).  The  figure  represents  the  supporting  Joint  Force  Operations 
of  TAC.  Each  organization  consists  of  personnel  and  equipment  forming 
functional  teams. 

An  in-depth  survey  of  each  organizational  team  is  not  feasible.  In  addition, 

the  problem  of  a  comprehensive  survey  is  magnified  further  by  the  existence 

2  2 

of  other  organizational  networks  of  C  systems  in  TAC,  C  networks  in 

other  commands  such  as  Strategic  Air  Command  (SAC)  and  Military 

2 

Airlift  Command  (MAC),  and  national  level  C  such  as  the  World-Wide 
Military  Command  and  Control  System  (WWMCCS)  and  the  National  Security 
Council/ Joint  Chiefs  of  Staff  (NSC/JCS). 

2 

The  C  systems  considered  for  inclusion  in  this  survey  are  presented  in 
Table  2.  Each  system  is  described  in  Appendix  A  to  this  volume.  The 
systems  considered  fall  into  three  categories:  current,  improvements 
to  current,  and  planned  or  future. 
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TABLE  1.  KEY  TO  ACRONYMS  IN  FIGURE  1 


ABCCC 

Airborne  Command  and  Control  Center 

AFCH 

Air  Force  Component  Headquarters 

ALCC 

Airlift  Command  and  Control 

ALCE 

Airlift  Control  Element 

ASOC 

Air  Support  Operations  Center  (formerly  DASC--  Direct 
Air  Support  Center) 

ASRT 

Air  Support  Radar  Team 

BDE 

Brigade 

BN 

Battalion 

CCT 

Combat  Control  Teams 

CRC 

Control  and  Reporting  Center 

CRP 

Control  and  Reporting  Post 

DIV 

Division 

FAC 

Forward  Air  Controller 

FACF 

Forward  Air  Control  Post 

JCS 

Joint  Chiefs  of  Staff 

JOC 

Joint  Operations  Command 

NCA 

National  Command  Authority 

SCAR 

Strike  Control  and  Armed  Reconnaissance 

TACC 

Tactical  Air  Control  Center 

TACP 

Tactical  Air  Control  Party 

woe 

Wing  Operations  Center  (formerly  TUOC--Tactical 

Unit  Operations  Center) 
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TABLE  2.  REPRESENTATIVE  AIR  FORCE  COMMAND  AND  CONTROL  SYSTEMS 


* 


1 


* 


i 
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I 


I 


» 
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2 

Several  sources  were  consulted  in  order  to  develop  this  set  of  C  systems, 
including  a  project  listing  for  the  Electronic  Systems  Division  (ESD)  of  the 
Air  Force  Systems  Command  (AFSC),  a  number  of  government  reports 
and  articles  in  defense  community  magazines,  the  Tactical  Air  Force 
Integrated  Information  System  (TAFIIS)  Master  Plan,  and  conversations 
with  C?  experts. 

The  number  of  systems  considered  had  to  be  reduced  to  make  this  research 

effort  manageable  and  compatible  with  the  time  and  resources  available. 

Command  and  control  systems  which  promised  high  payoff  in  the  form  of 
2 

significant  T  issues  were  identified  and  then  the  sample  was  narrowed  by 
selecting  representative  systems. 

2 

The  first  decision  concerned  the  identification  of  C  systems  which  would 
2 

yield  high- payoff  T  issues.  The  performance  requirements  for  strategic 
2 

and  tactical  C  systems  and  teams  are  different  owing  to  differences  along 

2 

three  situational  dimensions.  As  summarized  in  Table  3,  strategic  C 

is  characterized  by  specifiable  environmental  conditions,  predictable 

system  states,  and  available  solutions  and  probable  consequences.  On 

2 

the  other  hand,  tactical  C  is  characterized  by  unspecifiable  conditions, 

unpredictable  system  states,  and  less  available  options  and  probable 

2 

consequences.  This  suggests  that  strategic  and  tactical  C  teams  are 

different  in  structure  and  performance  requirements  and  thus  have 

different  training  needs.  The  informal  data  regarding  strategic  C" 

collected  during  this  effort  suggest  that  this  is  an  accurate  statement. 

The  more  established  nature  of  the  strategic  situation,  when  compared 

to  the  emergent  or  unexpected  nature  of  the  tactical  situation,  indicates 
2 

that  tactical  T  would  be  more  difficult  to  train.  Training  must  produce 
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TABLE  3.  CHARACTERISTICS  OF  STRATEGIC  AND  TACTICAL  C2 


Strategic 

Tactical 

Environmental 

Specifiable 

Not  specifiable 

Conditions 

States  of  the  System 

Predictable 

Not  predictable 

Probable  Consequences 

Available 

Potentially  available 

(Solutions) 

some  skills  that  are  generalizeable  to  a  large  set  of  possible  conditions, 
states,  options,  and  consequences.  The  difficulty  entails  producing  those 
skills  in  such  a  way  that  combat  readiness  can  confidently  be  assumed. 

2 

Improvements  in  T  can  be  expected  to  have  a  higher  payoff  in  tactical 
2 

C  systems  owing  to  this  emergent  nature  of  tactical  situations  and  the 

emphasis  on  real-time  decision  making.  Therefore,  it  was  proposed  to 

2 

l  concentrate  on  tactical  C  systems. 

I 

The  decision  to  focus  on  tactical  systems  excluded  not  only  strategic 
systems  including  Missile  Launch  and  SPADOC  but  also  systems  and 
programs  that  are  primarily  higher  organizations  for  analysis  and 
integration.  These  systems  and  programs  include  BETA  and  OASIS. 
NORAD  COC  itself  was  dropped  because  it  would  be  unproductively 
redundant  of  information  gathered  on  SAGE,  which  was  surveyed  as  a 
representative  part  of  the  North  American  Air  Defense  System. 


I 


i 


» 


i 


i 
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2 

Ultimately,  four  current  tactical/ air  defense  C  systems  formed  the  basis 
for  the  survey.  They  were  SAGE,  AWACS,  TSQ-91  Control  and  Reporting 
Center  (CRC),  and  TACC.  The  last  two  are  part  of  the  Tactical  Air 
Control  System  (TACS). 

2 

These  systems  are  representative  of  the  tactical/air  defense  C“  family 
and  functions-  The  SAGE  is  nearing  the  end  of  its  life  cycle,  the  TSQ-91 
is  mature,  and  the  AWACS  is  newly  developed.  The  SAGE  mission 
requires  daily  performance  of  tasks  and  procedures,  while  the  TSQ-91  is 
only  deployed  in  actual  combat.  The  AWACS  participates  in  actual  combat 
and  also  plays  a  role  in  other  operations  (such  as  rescue),  but  not  on  a 
daily  basis.  The  TACC  system  is  largely  manual  in  operation; 
computer  support  is  only  beginning  to  be  introduced. 

Finally,  in  order  to  ensure  that  recommendations  could  be  made  which 
2 

anticipate  future  C  systems  training  requirements,  five  systems  and 
subsystems  in  the  developmental  stage  of  acquisition  were  selected  to 
be  surveyed.  They  were  the  JSS--the  replacement  for  SAGE,  the  TIPI, 
the  TRI-TAC,  the  JTIDS,  and  the  System  Training  Exercise  Module 
(STEM)--an  improved  simulation  capability  for  the  TSQ-91.  In  addition, 
technological  developments  which  will  provide  advanced  system  capabil¬ 
ities  in  the  1990s  were  surveyed. 

Information  was  also  gathered  on  Red  Flag  and  Blue  Flag,  two  large-scale 
exercises  run  periodically  each  year.  The  purpose  of  the  exercises  is  to 
provide  experience  and  training  for  teams  in  tactical  air  warfare.  A  Red 
Flag  exercise  was  observed.  Blue  Flag  was  not  observed  as  a  result  of 
schedule  conflicts  and  availability  of  resources  to  do  adequate  observations. 


However,  the  facility  at  Eglin  AFB  was  visited  and  the  exercise  was 
discussed  with  some  individuals  associated  with  management  of  Blue 
Flag. 

The  primary  purpose  of  Red  Flag  is  to  exercise  fighter  pilots  in  air 
defense  against  a  Red  force  fighter  threat.  Live  flights  are  used  for 
both  sides  and  the  aggressor  force  uses  Red  tactics.  Ground-based 
intercept  controllers  participate  for  both  Blue  and  Red  forces.  AWACS 
participates  in  some  exercises  as  a  Blue  asset.  Observation  of  an 
AWACS  mission  was  the  purpose  of  our  visit.  Unfortunately,  AWACS 
did  not  fly  because  of  a  mechanical  failure.  Observations  were  made 
then  of  the  intercept  controllers  and  management  of  the  exercise. 

The  purpose  of  Blue  Flag  is  to  provide  experience  and  training  for 
personnel  of  a  Tactical  Air  Control  Center  of  the  Tactical  Air  Control 
Systems.  It  is  oriented  toward  staff  training  in  battle  management. 

A  numbered  Air  Force  is  the  primary  player;  there  are  no  radars  and 
no  live  flying.  The  TACC  is  fully  manned  and  exercised,  but  no 
lower-level  components  of  TACS  are.  Their  functions  on  the  effects  of 
the  activities  are  simulated.  The  operation  of  the  simulation  is  largely 
manual;  the  TACC  itself  is  primarily  manual  and  has  a  large  manual 
plotting  board  rather  than  consoles  with  CRTs  and  keyboards. 

The  sites  visited  during  the  survey  are  presented  in  Table  4.  The  data 
collection  methods  used  in  the  survey  are  discussed  in  Appendix  B  to  this 
volume. 
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TABLE  4.  SITES  VISITED  AND  RATIONALE 


Base/ Location 

Reason 

Luke /Phoenix,  Arizona 

TSQ-91  Weapons  Controller  training 

Tyndall /Panama  City,  Florida 

Weapons  Controller  basic  technical 
training;  manual  and  automatic  systems 

Keesler /Biloxi,  Mississippi 

Weapons  Technician  basic  technical 
training;  individual  training  plans 
and  development 

Tinker/Oklahoma  City, 

Oklahoma 

AW  ACS  T2 

Nellis/Las  Vegas,  Nevada 

Red  Flag  to  observe  AWACS  participation 

Duluth /Duluth,  Minnesota 

SAGE  T2 

Langley /Norfolk,  Virginia 

■>  2 

HQ  TAC  C”T”  planning 

I 

Eglin/ Pensacola,  Florida 

•> 

TSQ-91  T"  and  TACS  test  bed 

Shaw /Sumter,  South  Carolina 

TACC  T2 

Hans  com  /  Bedford, 

Massachusetts 

> 

ESD  C”  system  design  and 
acquisition 

The  remainder  of  this  volume  contains  a  brief  review  of  relevant  literature 
and  technical  documentation,  results  of  interview  of  training  program 
personnel,  and  a  summary  of  observations  of  training. 

DEFINITIONS  AND  FRAMEWORK  FOR  ANALYSIS 

There  is  a  need  to  establish  a  common  ground  by  defining  how  the  terms 
command  and  control,  and  team  training,  will  be  used  in  this  report.  Also 
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the  term  Instructional  System  Development  (ISD)  will  be  defined  because 

it  was  used  as  a  framework  for  discussing  definition,  development, 

2  2 

implementation,  and  evaluation  of  AFC  T  programs.  Finally,  a 

2  2 

preliminary  model  for  considering  AFC  T  is  presented  as  an 
organizing  element  for  the  remainder  of  this  report. 

2 

Command  and  Control  and  C  Systems 

The  DoD  Dictionary  of  Military  and  Associated  Terms  (JCOS  Pub  1) 

,  ..  _2 
defines  C  as: 

The  exercise  of  authority  and  direction  by  a  properly  designated 

commander  over  assigned  forces  in  the  accomplishment  of  his 
2 

mission.  C  functions  are  performed  through  an  arrangement 
of  personnel,  equipment,  communications,  facilities,  and 
procedures  which  are  employed  by  a  commander  in  planning, 
directing,  coordinating,  and  controlling  forces  and  operations 
of  his  mission. 

3  3 

Many  variations  of  nomenclature  exist  such  as  C  and  C  I  for  referring 
2 

to  C  but  the  additional  functions  are  redundant  with  the  above  definition. 
2 

Therefore,  C  is  used  in  this  report  to  encompass  all  these  terms. 

Generally,  a  system  is  an  arrangement  of  things  which  are  related  to 

form  a  unity.  In  a  military  context,  the  things  in  the  arrangement  may 

be  personnel,  equipment,  units,  etc,  or  a  combination  of  these.  Air 
2 

Force  C  systems  are  complex,  organized  entities  composed  of  many 
such  elements.  For  example.  Figure  1  depicts  the  Air  Force  system 
which  supports  joint  force  operations.  (The  acronyms  in  the  figure  are 


decoded  in  Table  1. )  Each  center,  post,  unit,  or  aircraft  depicted  in 

Figure  1  is  itself  a  system,  or  more  properly,  a  subsystem  of  the  tactical 

operations  structure.  Each  subsystem  consists  of  personnel  equipment 

2 

organized  to  allow  certain  C  functions  and  tasks  to  be  carried  out. 

Take,  for  example,  the  Control  and  Report  Center  (CRC)  of  the  Tactical 

Air  Control  System  (TACS).  A  typical  CRC  consists  of  approximately  90 

people,  a  hardware /software  system  known  as  the  TSQ-91  (407L),  and 

other  equipment.  The  CRC  is  organized  into  roughly  four  functional  areas: 

surveillance,  identification  (both  acquisition  and  information),  command/ 

battle  staff  (decision  making),  and  weapons  control  (implementation). 

2 

This  type  of  C“  team  structure  is  illustrated  in  Figure  2.  Each  of  these 

i 

functions  is  performed  by  a  number  of  personnel  organized  in  a  section 
which  is  supported  by  equipment  or  information.  The  essential  components 
i  of  a  TSQ-91  are  consoles  which  display  radar  imagery  and  the  personnel 

who  observe  it  to  determine  the  location,  heading,  and  speed  of  aircraft 
and  missiles. 

2 

The  CRC  thus  functions  to  a  large  degree  in  a  real-time  C  mode. 

4  Individuals  performing  the  functions  noted  above  in  the  CRC  and  other 

2 

tactical  air  C  systems  make  complex  decisions  under  high  stress.  This 
report  is  about  the  performance  required  of  such  individuals,  especially 
*  in  a  team  context  as  defined  below,  and  how  those  individuals  and  teams 

are  trained. 
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Figure  2.  Generic  Structure  of  Air  Force  Tactical  or  Air 
Defense  Command  and  Control  System 

Teams:  Performance  and  Training 


I 


i 


A  team  is  a  collection  of  individuals  engaging  in  cooperative  activities  to 
achieve  a  common  goal.  Teams  may  be  partly  distinguished  from  other 
goal-oriented  groups  because  teams  are  "usually  well  organized,  highly  r 

structured,  and  have  relatively  formal  operating  procedures"  while  other 
small  groups  are  "rarely  so  formal  with  such  well  defined,  specialized 
tasks"  (Reference  1).  While  the  distinction  between  teams  and  small 
groups  may  be  made  on  the  basis  of  formal  structure  and  procedure,  it 
is  important  not  to  overlook  performance.  Several  teams  with  the  same 
structure  and  procedures  may  greatly  differ  in  the  achievement  of 
mission  goal?  and  objectives.  By  structure  and  procedure  they  are 
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2 

equivalent  teams;  in  practice  they  are  not.  C  teams  operate  in  a  complex, 
stressful  environment  in  which  performance  is  a  function  of  interaction 
among  multiple  factors  and  components.  The  processes  by  which  the  more 
productive  teams  function  are  more  adaptable  and  robust.  Therefore  an 
adequate  definition  of  a  team  must  include  both  structural  and  process 
parameters. 

Meister  (Reference  2)  includes  an  element  of  performance  in  his  description 
of  team  characteristics.  He  identifies  three  primary  characteristics  of 
teams: 

•  Relatively  rigid  structures  with  formal  organization  and 

t  communication  networks 

•  Well-defined  positions  or  member  assignments 

•  Cooperation  or  coordinated  participation  of  several 
specialized  individuals  whose  activities  contain  little 
overlap 

In  general,  any  man-machine  system  containing  two  or  more  operators 
working  in  interaction  involves  a  team.  Moreover  team  performance  is 
often  described  as  a  unitary  phenomenon.  That  is,  the  team,  rather  than 
the  individuals,  works  to  a  mission  requirement,  performs  tasks, 
receives  feedback,  and  adjusts  its  behavior  to  changing  demands 
(Reference  2). 

Research  on  the  team  is  usually  based  on  the  assumption  that  there  is 
some  element  of  team  performance  which  is  somehow  different  from  the 
collective  performance  of  a  set  of  independent  individuals.  Analytic 
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research  suggests  that  the  interaction  between  team  members  is  the 
critical  element  of  team  performance.  The  simplicity  and  generality  of 
this  suggestion  makes  it  both  appealing  as  an  explanation  and  difficult  to 
prove  or  disprove.  The  types  of  team  training  which  it  suggests  (for 
example,  communication,  coordination)  are  necessary  but  probably  not 
sufficient  to  achieve  a  high  level  of  team  performance. 

Team  Performance  /  Training  Mod  el- -Team  training  cannot  be  adequately 
discussed  without  a  consideration  of  team  structure  as  the  context  for 
performance.  A  complete  discussion  must  also  include  an  analysis  of 
the  specific  military  training  which  prepares  personnel  for  team  positions. 

I 

Brock  (Reference  3)  has  proposed  a  model  which  incorporates  team  context 
and  performance  with  levels  of  training.  The  model  breaks  the  team  into 
units  which  are  meaningful  in  both  performance  and  training,  and  shows  1 

where  the  particular  team  is  directly  related  to  other  elements  in  the 
system.  The  model  has  proved  to  be  a  useful  analytic  tool  in  application 
(Reference  4). 

Four  levels  of  performance,  context,  and  training  are  established:  * 

individual,  subteam,  team,  and  superteam.  The  levels  of  training  are 
illustrated  in  Figure  3  for  the  position  of  weapons  controller  in  a  CRC/CRP. 

\ 

Positional  duties  and  interface  functions  with  other  positions  are  learned 
during  individual  training.  Combinations  of  individuals  are  brought  together 
during  the  subteam  level  to  develop  crews  within  functional  areas  such  as 
weapons  direction,  surveillance,  battle  staff,  and  other  areas.  The  team 
training  level  is  devoted  to  integrating  these  crews  within  an  organization 
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into  a  team  capable  of  accomplishing  the  mission  of  the  organization  such 
as  a  TACC,  CRC/CRP,  ASRT,  and  FACP.  Finally,  the  superteam  level 
consists  of  multi-organizational/multi-team  exercises  which  involve  more 
than  one  organization  within  a  command,  from  two  or  more  commands 
and  multi-service. 

This  model  is  not  necessarily  how  training  is  actually  done  or  defined  in 
any  organization;  rather,  it  is  a  statement  of  the  conditions  for  adequate 
training  to  occur  based  on  observation  and  analysis  of  teams  and  training. 

A  further  description  of  the  team  performance /training  model  will  be 
presented  and  an  example  from  Air  Force  command  and  control  will  be 
developed  further.  The  example  is  running  an  intercept.  The  team 
running  the  intercept  consists  of  the  pilot  and  the  ground  personnel  doing 
surveillance  and  weapons  direction. 

Individual  Performance  and  Training--Performance  at  the  individual  level 
is  focused  on  the  individual  operator  and  behavior  is  described  for  the 
uniquely  individual  task  elements.  Each  member  of  a  team  has  a  set  of 
individual  skills  which  must  be  performed  if  the  team  is  to  successfully 
operate  as  an  integrated  unit.  An  aircraft  pilot,  for  example,  must  have 
a  large  set  of  complex  skills  in  order  to  fly  today's  advanced  aircraft. 

Training  at  the  individual  level  should  be  more  inclusive  than  just  how  to 
perform  a  task.  Hie  pilot  should  have  information  not  only  on  how  to 
perform  the  individual  task  elements  and  practice  at  doing  so,  but  also 
should  have  information  on  where  those  task  elements  fit  into  the  larger 
team  context. 


Subteam  Performance  and  Training — Performance  at  this  level  is  focused 
on  subsets  of  the  larger  team,  for  which  proficiency  in  recurring  interac¬ 
tions  is  a  necessary  component  of  effective  team  performance.  Training 
at  this  level  should  consist  of  the  development  of  unique  subteam 
performance  skills.  The  full  set  of  individual  skills  for  the  particular 
position  and  knowledge  about  the  subteam  are  requisite  to  this  level. 

It  is  usually  possible  to  identify  several  subteams  within  a  team.  During 
an  intercept  it  is  necessary  for  the  weapons  controller  and  the  pilot  to 
interact  on  a  recurring  basis  so  that  the  aircraft  is  maneuvered  close 
enough  to  the  hostile  to  pick  it  up  on  the  pilot's  airborne  radar.  Both  the 
pilot  and  the  weapons  controller  must  have  a  "feel"  for  the  hostile's 
tactics,  the  characteristics  of  the  aircraft,  and  the  tactical  situation  so 
that  they  may  select  an  intercept  course,  anticipating  events  and  leading 
the  target  appropriately. 

Team  Performance  and  Training- -Performance  at  the  team  level  is 
focused  on  the  immediate  set  of  individuals  tasked  with  a  common  goal. 
They  are  the  primary  team.  They  ir  y  or  may  not  share  a  common 
location,  but  they  are  always  linked  somehow.  In  an  intercept  one  might 
consider  the  pilot,  weapons  controller,  and  weapons  controller  technician 
to  be  the  relevant  team. 

The  team  has  a  set  of  objectives  and  an  overriding  goal.  These  objectives 
and  goals  are  usually  stated  at  the  team  level.  Team  performance  is 
described  only  when  it  is  evaluated  against  these  objectives  and  goals. 
However,  performance  is  most  easily  observed  at  the  individual  and 
sub  team  levels. 
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Training  at  this  level  should  involve  the  development  and  practice  of 
unique  team  performance  skills.  Usually  this  means  the  integration  of 
individual  and  subteam  skills  in  a  real  or  realistic  environment. 

Superteam  Performance  and  Training  —  Performance  is  focused  at  the 
team  level  but  now  includes  interaction  with  those  friendlies  who  direct 
or  are  directed  by  the  team.  This  interaction  may  establish  goals,  be 
directed  to  achieving  specific  objectives,  or  request  or  provide  information. 
In  an  intercept  one  might  consider  the  superteam  to  consist  of  the  pilot, 
weapons  controller,  and  weapons  controller  technician  (the  immediate 
team),  together  with  surveillance  personnel  and  the  weapons  assignment 
officer.  Surveillance  initially  picks  up  the  aircraft  and  identifies  it  as 
friendly  or  hostile.  If  hostile,  the  weapons  assignment  officer  is  informed 
and  a  weapons  controller  is  assigned  to  handle  the  intercept. 

Training  at  this  level  should  involve  the  further  integration  of  the  team 
into  the  framework  in  which  it  will  function  during  actual  combat. 

Referring  to  Figure  1,  the  CRC,  for  example,  would  exercise  in  support 
of  a  TACC  and  simulate  communications  with  it  and,  perhaps,  the  AWACS, 
a  CRP,  or  FACP.  If  the  TACC  were  the  unit  in  question,  then  superteam 
elements  would  include  AFCH,  ALCC,  CRC,  AWACS,  ASOC,  WOC,  and 
so  on  (see  Table  1  for  acronym  meanings). 

This  integration  of  individual  and  team  skills  in  a  real  or  realistic 
environment  depends  upon  the  prior  development  of  individual,  subteam, 
and  team  skills,  together  with  knowledge  about  the  larger  framework 
within  which  the  team  will  perform. 


This  discussion  shows  how  the  structure  and  performance  of  a  group  of 
persons  with  a  common  set  of  goals  and  objectives  is  conceptually  broken 
down  into  individual,  subteam,  team,  and  superteam  components. 

*  Training  can  be  designed  to  achieve  specific  performance  objectives 

within  each  of  these  components. 

The  process  of  dividing  the  group  into  those  components  is  not  well  defined. 
The  boundaries  drawn  during  analysis  are  somewhat  arbitrary  and  depend 
on  such  factors  as  the  purpose  and  time  scale.  Moreover,  a  set  of 
criteria  do  not  exist  which  allow  the  observer  to  decide  whether  the 
divisions  are  appropriate.  However,  the  model's  flexibility  and  ease 
of  use  make  it  a  useful  analytic  tool  despite  its  several  shortcomings. 

i 

Extension  of  the  Team  Training  Model/ Performance 

i 

Two  dimensions  of  personnel  category  and  training  program  type  will  now 
be  added  to  the  four  levels  of  team  performance  context.  Figure  4  shows 
these  dimensions  and  the  categories  within  each  of  them.  The  following 
discussion  on  personnel  is  based  on  an  air  defense  mission  such  as  that 
4  found  in  a  TACS.  Similarly,  the  types  of  training  programs  are  based 

on  the  practices  and  regulations  in  TAC. 

2 

'  Personnel  Category — There  are  three  personnel  categories  in  C  systems: 

console  operators  and  support  personnel,  first-line  supervisors,  and 
battle  staff. 
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Console  operators  and  support  personnel  include  weapons  controllers  / 
directors;  weapons  technicians;  surveillance,  tracking,  and  height 
technicians;  plotters,  tellers,  and  recorders;  and  airborne  radar 
»  technicians,  radio  operators,  and  computer  maintainers  and  operators. 

These  individuals  perform  functions  which  involve  the  control  of  individual 
aircraft  or  the  development  of  information  and  data  which  reveal  the 
state  of  the  battle. 

First-line  supervisors  consist  of  weapons  directors,  senior  controllers, 

V 

surveillance  officers,  radar  inputs  countermeasures  officers,  and  height 
and  tracking  noncommissioned  officers.  These  individuals  manage  people 
\  and  air  resources  under  their  immediate  control. 

Finally,  there  are  the  personnel  of  the  battle  staff,  consisting  of  the 
;  commander  and  his  assistants  in  planning,  operations,  logistics, 

intelligence,  and  so  on.  These  individuals  manage  information  and 
resources. 

Training  Program  Type- -Air  Force  training  is  accomplished  in  both 
*  institutional  and  operational  settings.  As  a  general  rule,  institutional 

training  is  the  responsibility  of  Air  Training  Command  and  operations 
training  is  the  responsibility  of  the  operational  command;  Tactical  Air 
'  Command  in  this  case.  Institutional  training  is  formal  and  involves 

courses  with  a  specific  curriculum  and  length.  Operations  training  is 
typically  informal  and  training  is  on-the-job.  There  may  also  be  some 
formal  classroom  instruction  and  there  are  self-study  programs. 


23 


■v? 


Air  Force  training  consists  further  of  a  progression  through  four  levels 

designed  to  1)  impart  initial  fundamental  knowledge  and  skills  in  a 

particular  career  field  and  specialty;  2)  transition  those  knowledges 

and  skills  to  an  operational  environment;  3)  provide  for  continuation 

(upgrade,  advance,  or  maintenance)  of  knowledges  and  skills;  and 

4)  exercise  those  knowledges  and  skills  under  conditions  which  attempt 

to  simulate  actual  job  conditions  as  closely  as  possible  (Reference  5). 

The  survey  indicated  that  an  institutional  setting  was  characteristic  of 

initial  training,  the  first  phase  of  transition  training  and  certain  advanced 

training  within  the  continuation  category.  All  other  training,  including 
2 

most  T  ,  is  conducted  in  an  operational  setting. 

The  need  for  initial  training  of  console  operator  and  support  personnel 
is  well  accepted.  Table  5  is  a  list  of  courses  which  exist  for  these 
purposes.  The  performance  context  in  these  programs  is  primarily 
individual  but  subteam  contexts  such  as  controller-technician- pilot  are 
also  fundamental  to  this  training. 

The  first  phase  of  transition  training  of  console  operators  and  support 

personnel  has  been  formalized  into  an  institutional  format  for  the  three 
2 

C  systems  surveyed.  The  training  responsibility  shifts  from  Air 
Training  Command  to  TAC.  In  the  case  of  SAGE  this  training  occurs 
in  the  units.  Both  the  CRC  and  AWACS  have  training  squadrons  whose 
sole  mission  is  tc  prepare  console  operators  for  the  transition  from 
classroom  to  an  operations  setting.  The  607th  Tactical  Control  Training 
Squadron  (TCTS)  accomplishes  this  for  the  TACS  CRC  and  the  966th 
AWACS  Training  Squadron  for  AWACS.  The  primary  performance  contexts 
of  this  type  of  training  axe  individual  and  subteam;  team  contexts  are  used 
by  some,  but  not  all,  personnel. 
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TABLE  5.  AFC  INITIAL  TRAINING 
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Training  in  operations  settings  is  informal.  In  general,  first-line 
supervisors  and  battle  staff  are  trained  in  team  and  superteam  contexts 
primarily  in  exercise  programs.  Hie  exercises  vary  in  scale  and  may 
be  local  or  involve  other  units.  Blue  Flag  is  an  example  of  a  large-scale 
exercise  for  the  TACS  TACC. 

Instructional  System  Development  (ISP) 

Instructional  System  Development  is  sin  attempt  to  provide  and  standardize 
a  systematic  method  for  developing  and  managing  training  programs. 

Team  training  can  be  incorporated  more  readily  into  the  existing  procedures 
if  its  analysis  and  development  are  at  least  compatible  with  ISD.  At  any 
rate,  ISD  provides  a  preliminary  framework. 

Training  programs  have  a  life  cycle  consisting  of  three  stages:  definition 
and  development,  implementation,  and  evaluation.  These  three  stages 
can  be  systematically  and  effectively  carried  out  through  the  application 
of  ISD,  which  details  the  steps  and  analytic  techniques  to  be  followed  and 
applied. 

The  use  of  ISD  is  required  by  regulation.  However,  there  are  differences 
among  the  armed  services  on  the  ISD  model  and  procedures.  Each  service 
has  developed  its  own  model  and  procedures.  There  is  also  an  interservice 
model  developed  jointly  by  the  Army  and  Navy  known  as  ITRO  (Reference  6). 
The  Air  Force  prescribes  the  use  of  ISD  "to  plan,  develop,  and  manage" 
all  training  and  educational  programs  whether  new  or  modified  (AF 
Regulation  50-8).  Regulation  50-8  (Reference  7)  specifies  the  use  of 
AFM  50-2  (Reference  8),  AFP  50-58  (Reference  9),  or  ITRO  for 
accomplishing  ISD. 
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There  are  several  drawbacks  and  inadequacies  in  existing  ISD  methods. 
There  is  no  guarantee,  for  example,  that  the  application  of  two  different 
ISD  models  to  the  same  training  problem  will  result  in  identical  programs 
*  because  the  procedures  differ  slightly  or  are  more  or  less  well-specified 

from  any  one  model  to  another.  Also,  the  same  model  applied  by  two 
different  institutions  or  groups  may  not  yield  identical  programs. 

These  points  illustrate  that  although  the  ISD  technique  is  well-formulated, 
it  is  largely  heuristic  in  its  application.  Therefore,  the  particular  ISD 
model  used  and  the  institution  or  group  developing  instruction  are  factors 
bearing  on  the  effectiveness  of  the  definition  and  development  process. 

In  order  to  avoid  such  problems,  we  need  to  adopt  a  standard  which,  if 
not  algorithmic,  is  at  least  complete  in  specifying  all  necessary  steps 

if 

to  be  performed. 

.  Vineberg  and  Joyner  (Reference  10)  have  developed  a  generalized  model 

of  the  ISD  process.  Based  on  an  analysis  of  the  service  models  and 

ITRO,  their  ISD  model  specifies  a  set  of  19  steps.  Table  6  presents 

their  model  and  shows  how  the  steps  may  be  segregated  into  four  phases: 

assessment  of  training  need,  analysis  of  training  requirements,  transition 

4  from  requirements  to  implementation,  and  program  evaluation.  These 

phases  of  the  ISD  process  parallel  the  organization  of  the  subsequent 

2  2 

chapters  of  this  volume  in  which  the  status  of  AFC  T  is  characterized. 

The  purpose  in  adopting  an  ISD  framework  was  to  provide  a  set  of  rubrics 
for  planning  the  analysis  of  training  programs,  data  gathering,  reporting 
results,  and  discussing  them.  There  are  problems  and  deficiencies  with 
ISD  procedures  but  these  are  largely  at  levels  below  which  ISD  is  being 
used  in  this  study.  Some  of  the  problems  have  been  discussed.  Another 
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TABLE  6.  STEPS  OF  A  GENERALIZED  ISD  MODEL 

(After  Vineberg  and  Joyner,  1980--Reference  10) 


• 

Assessment  of  training 

Needs 

• 

Identification  of  job  requirements 

• 

Selection  of  tasks  for  training 

Training 

Requirements 

• 

Analysis  of  tasks 

Analysis 

# 

Construction  of  job  performance  measures 

• 

Selection  of  setting 

• 

Development  of  training  objectives 

• 

Development  of  achievement  tests 

• 

Identification  of  entry  knowledge /skill  levels 

• 

Classification  of  objectives  and  selection  of 
instructional  activities 

• 

Selection  of  instructional  methods 

Transition  from 

• 

Selection  of  media 

Requirements  to 
Specific  Program 

• 

Grouping  and  sequencing  of  instruction 

• 

Development  of  plan  for  authoring  and  managing 
instruction 

• 

Review  and  selection  of  existing  materials 

• 

Authoring  of  instruction 

• 

Validation  of  instruction 

• 

Internal  evaluation 

Program 

• 

External  evaluation 

Evaluation 
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significant  problem,  is  that  the  procedures  are  not  satisfactory  for 
describing  cognitive  or  semi-structured  tasks  in  emergent  situations; 
they  work  best  for  fixed  procedural  tasks  in  established  situations.  The 
population  of  tasks  of  teams  is  heavily  weighted  with  cognitive  semi- 
structured  tasks.  Furthermore,  ISD  does  not  provide  procedures  for 
deriving  features  of  simulation.  However,  the  intended  use  of  ISD  in 
this  study  does  not  include  doing  detailed  task  descriptions  or  analysis. 

Its  use  is  intended  to  provide  a  framework  which  is  consistent  with 
existing  practices  in  the  training  community. 

The  Role  of  ISD  in  Team  Training  Tactical  Air  Command 

A  first  approximation  to  a  generalization  about  the  role  of  ISD  is  that  it 
is  used  in  some  form  for  formal,  institutional  training  courses  but  is  not 
used  for  developing  informal,  operational  training  and  exercises.  ISD 
is  used  to  define  training  requirements  and  objectives  by  analysis  of  job 
performance  requirements,  to  prepare  training  materials,  and  to 
evaluate  the  training  program. 

The  courses  for  initial  training  listed  in  Table  5  were  developed  in  this 
manner.  Some  courses  which  antedate  ISD  were  restructured  and  ISD- 
justified  once  the  method  was  adopted.  The  TAC  courses  for  transition 
training  into  SAGE,  CRC,  and  AWACS  were  also  developed  by  ISD. 

In  contrast,  informal  and  on-the-job  training  (OJT)  in  operational  settings 
has  little  formal  structure.  It  emerges  out  of  the  events  and  problems 
that  occur  in  exercises.  Hie  exercises  vary  in  scope  over  single  units. 
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multiple  units,  and  the  complexity  of  Blue  Flag  exercises  and  JCS  exercises 
such  as  Gallant  Eagle,  Brave  Shield,  and  Bold  Eagle. 


Training  needs  and  training  objectives  do  not  appear  to  get  explicit 
consideration.  Console  operators  and  support  personnel  might  be  able  to 
get  credit  for  some  STAN/EVAL  requirements  but  these  requirements  are 
usually  met  during  supervised  exercises  within  the  squadron.  First-line  1 

supervisors  and  battle  staff  members  appear  to  be  trained  entirely  during 
exercises  at  the  scale  of  team  and  superteam  levels. 

The  design  and  development  of  exercises,  whatever  their  scale,  do  not 

follow  anything  approximating  an  ISD  procedure.  Training  requirements 

and  objectives  are  rarely  defined.  Feedback  is  informal  and  unsystematic.  * 

After-action  reports  (AARs)  are  written,  endorsed,  and  filed.  We  found 

no  procedure  to  ensure  their  use  in  planning  subsequent  exercises  or  to  4 

ensure  that  they  be  used  in  a  formal  and  systematic  manner. 

The  procedure  for  preparing  exercises  which  was  described  by  our 

respondents  is  based  on  a  committee  of  representatives  from  the 

participating  units  and  consultants.  The  participating  units  might  be  a 

CRC/CRP,  TACC,  and  Wing  headquarters.  The  consultants  are  exercise 

developers  under  a  support  contract.  The  committee  discusses  what  it 

can  and  would  like  to  do  and  reaches  a  consensus.  The  consultants  then  t 

prepare  the  scenario,  tapes  of  simulated  imagery,  and  scripts. 
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The  ostensible  purpose  of  the  exercise  is  often  something  other  than 
training.  One  exercise  that  we  observed  was  held  for  the  purpose  of 
evaluating  a  communication  switch  under  development.  The  organizations 
•  involved  were  two  CRPs,  a  CRC,  and  a  TACC.  We  observed  in  the  CRC 

during  the  first  day.  Squadron  personnel  observed  performance  of 
individuals,  took  notes,  and  debriefed  the  unit.  We  observed  in  the 
TACC  during  the  second  day  and  were  present  for  the  planning 
teleconference  of  all  participants  at  the  beginning  of  the  second  day.  The 
main  business  of  the  teleconference  was  changes  in  the  exercise  to  get  a 
better  test  of  the  communication  switch.  There  was  no  discussion  of 
training  needs  or  objectives. 

» 

Red  Flag  is  explicitly  a  training  exercise  for  fighter  pilots.  Planning  for 
an  exercise  begins  45  days  prior  to  its  scheduled  date.  Training  needs 
.  of  the  pilots  are  given  more  consideration,  perhaps  in  part  because  the 

exercise  is  run  by  pilots  to  serve  the  needs  of  pilots.  Still  the  exercise 
is  planned  by  committee  and  the  procedure  is  informal,  unsystematic  and 
non-ISD.  Training  for  ground  controllers  and  AWACS  personnel  in  Red 
Flag,  however,  is  a  relatively  unstructured,  serendipitous  byproduct. 

« 

OVERVIEW  OF  THE  PRESENTATION  OF  SURVEY  RESULTS 
2  2 

'  The  status  of  AFC  T  is  characterized  in  the  succeeding  chapters  of  this 

volume.  Chapter  II  describes,  analyzes,  and  evaluates  the  definition  and 

2  2  o  9 

development  of  AFC  T  .  Similarly,  Chapter  HI  deals  with  AFC  T 

implementation  and  management,  and  Chapter  IV  with  program  evaluation 

and  modification.  Within  each  chapter  the  topic  of  simulation  is  given 


co-equal  status  with  training  program  issues.  Simulation  is  a  key  element 
2  2 

of  C  T  because  all  training  is  carried  out  under  simulated  conditions  as 

compared  to  actual  combat.  Even  training  that  involves  live  flying  is 

conducted  under  conditions  of  simulated  combat  condition.  ISD  does  not  . 

treat  simulation  beyond  its  potential  selection  as  an  appropriate  training 

2  2 

medium.  This  is  a  weakness  of  ISD  as  applied  to  AFC  T  .  Chapter  V  of 

2  2  t 

this  volume  summarizes  the  strengths  and  weaknesses  of  AFC  T  and 
identifies  issues  and  problem  areas  in  need  of  resolution. 

2  2 

SUMMARY  OF  STRENGTHS  AND  WEAKNESSES  OF  AFC  T 

The  results  of  the  interviews  and  observations  are  summarized  into 

two  sets  of  generalizations  stating  the  strengths  and  weaknesses  of 

AFC  T  .  They  are  organized  into  the  categories  of  definition  and 

development,  implementation  and  management,  and  evaluation  and  , 

modification,  representing  the  major  phases  of  the  life  cycle  of  a 

training  program. 

The  analyses  during  the  study  deal  predominantly  with  the  weaknesses 

since  a  purpose  of  the  study  was  to  recommend  research  and  development 

projects  that  will  contribute  to  improving  AFC2T2 .  The  weaknesses 

were  condensed  into  a  set  of  underlying  issues  and  problem  areas  which 

will  be  discussed  soon.  The  recommendations  were  derived  from  the  t 

issues  and  problem  areas. 
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Strengths 


Definition  and  Development- - 
2 

•  C  system  design  .documents,  for  example,  data  items,  provide 
ample  opportunity  for  incorporating  good  human  factors. 

2  2 

•  AFC  T  personnel  are  dedicated  and  motivated  and  perform 
admirably  considering  the  limited  resources  they  have. 

2  2 

•  Recognition  of  the  critical  importance  of  C  T  for  joint 
service  operations  is  increasing. 

•  Systematic  procedures  within  Instructional  System  Development 
(ISD)  exist  for  identifying  individual  operator  job  tasks  and  task 
elements,  especially  as  related  to  the  equipment  console. 

2  2 

•  The  importance  of  simulation  in  accomplishing  C  T  is  well- 
accepted. 

•  The  development  of  course  control  documents,  for  example,  course 
training  standards,  curriculum  lesson  plans,  etc. ,  provides 
formal  training  objectives  for  individual  operator  initial  and 
advanced  training. 

•  The  System  Training  Plan  developed  for  new  systems  is  a  com¬ 
prehensive  instructional  management  and  production  plan. 
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Implementation  and  Management- - 

•  Individual  operator  skills  are  systematically  trained;  difficulty 
is  sequenced  effectively. 

•  The  limited,  predictable  nature  of  static  air  defense  missions 
makes  their  training  relatively  straightforward;  system 
training  exercises  for  SAGE  are  well-defined. 

•  The  standardization/ evaluation  program  is  clearly  successful 
in  monitoring  individual  skills  readiness. 

•  Intrateam  briefings  addressing  issues  of  intra-  and  inter- section 
performance  are  helpful  in  building  team  awareness  and  a  shared 
plan. 

Evaluation  and  Modification — 

•  The  independent  (from  a  specific  training  program)  status  of 
the  Tactical  Air  Control  System  Office  of  Training  Development 
is  necessary  for  meaningful  program  evaluation. 

•  The  standardization/evaluation  program  prevents  major 
discrepancies  in  T^  programs. 

•  Questionnaire  approaches  are  positive  attempts  to  query  students 
and  operational  units  regarding  the  quality  of  training  programs. 

2 

•  Expert  judgment  is  essential  to  the  evaluation  of  T  programs, 
especially  large-scale  exercises  like  Blue  Flag. 
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The  strengths  show  good  programs  for  individual  training  and  the  existence 

of  the  mechanisms  and  practices  necessary  for  developing,  conducting, 

and  evaluating  training.  Throughout  the  survey  it  was  evident  that 
2 

AFC  training  personnel  were  concerned  about  improving  the  effectiveness 
2 

of  T  .  The  picture  emerged  of  competent,  dedicated  individuals  doing 

the  best  they  could  with  limited  resources  and  technologies.  There  is 

2  2 

a  recognition  of  the  weakness  of  C  T  and  a  desire  to  take  positive, 
corrective  action. 

Training  for  individual  operators  is  effectively  developed  and  implemented. 
The  assessment  of  operators'  individual  skills  is  comprehensive  and 
well-standardized.  Efforts  to  build  team  identity  and  spirit  through 
intrateam  briefings  are  good. 

Weaknesses 


Definition  and  Development — 

•  Enforcement  of  human  factors  data  requirements  is  incomplete 
or  lax  as  a  result  of  pressures  to  develop  the  system. 

•  Training  is  often  incorrectly  viewed  as  a  panacea  for  ineffective 
system  design. 

•  Joint  services  system  acquisition,  for  example,  TRI-TAC,  put 
stress  on  training  developers  who  might  have  to  contend  with 
different  training  philosophies  and  personnel  constraints. 

2  2 

•  In  competition  for  scarce  or  limited  resources,  the  C  T  area 
has  historically  had  a  low  priority  in  the  Air  Force. 
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•  There  are  no  systematic  procedures  for  defining  the  appropriate 

2 

team  structure  for  a  C  system  or  for  allocating  tasks  to  the 
team  as  opposed  to  the  individual  operator  or  system  software. 

* 

•  Instructional  System  Development  (ISD)  techniques  do  not 

2 

address  team  tasks,  skills,  or  T  objectives,  nor  do  they 
adequately  address  non-console  tasks  and  the  tasks  of  non- 
operators . 

2 

•  There  are  no  systematic  procedures  for  defining  C  system 

simulator  requirements;  nor  is  there  empirical  data  on  the 

2  2 

level  of  fidelity  required  for  C  T  . 

2 

•  The  procurement  of  C  systems  radically  different  from  existing 
systems,  for  example,  AWACS,  puts  pressure  on  training  developers 
because  "job"  experts  find  it  difficult  to  relate  to  unfamiliar 

operating  procedures.  * 

2 

•  The  definition  of  T  requirements  is  hampered  by  a  lack  of 

2 

articulation  of  what  constitutes  proficient  C  team  performance. 

•  Training  requirements  for  operations  training  programs  (in- unit) 

are  of  dubious  validity  because  they  have  not  been  related 

2  * 

empirically  to  C  skills  achievement  and  maintenance. 

2 

•  Operations  training  program  requirements  for  live  T  are 

constrained  by  the  availability  of  flying  resources.  * 

•  Training  objectives  are  developed  informally,  if  at  all,  for 
transition  and  continuation  training  in  the  context  of 
simulated  combat  missions /system  exercises. 
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•  The  selection  of  instructional  methods,  media,  and  sequencing 
is  primarily  influenced  by  factors  unrelated  to  training 

,  effectiveness;  namely,  tradition,  resource  constraints,  etc. 

•  Unified,  comprehensive  plans  dealing  with  the  management 
and  production  of  operations  training  do  not  exist. 

Implementation  and  Management- - 

•  There  are  no  minimum  aptitude  requirements  for  entry  into  the 
Air  Weapons  Controller  career  field. 

•  Team-oriented  skills  are  not  trained  systematically;  sequencing 
of  training  is  not  optimized. 

2 

•  There  is  no  formal,  standardized  training  for  C  system 
supervisors . 

•  The  emergent  nature  of  tactical  missions  makes  tactical  skill 
training  difficult;  simulated  combat  missions /system  exercises 
for  AWACS  and  TACS  are  ill-defined. 

•  The  evaluation  of  team  readiness  is  hampered  by  a  lack  of 

i  articulation  of  the  dimensions  and  attributes  of  good  team 

performance. 

•  Team  readiness  assessment  is  not  standardized  in  terms  of 

* 

conditions  and  difficulty. 

•  The  importance  of  feedback  to  teams  is  recognized  but  there 
are  no  proven  techniques  for  making  feedback  effective. 
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•  Simulators  do  not  fully  support  instructor  requirements. 

•  Simulation  fidelity  does  not  support  effective  training. 

•  Personnel  who  simulate  interceptor  pilots  are  not  formally 
trained  for  their  duty. 

2 

•  C  career  fields  are  characterized  by  low  retention  rates 
and  the  steady  loss  of  experienced  individuals. 

2 

•  C  training  program  managers  have  an  incomplete  understanding 

2  2 

of  the  C  T  pipeline. 

Evaluation  and  Modification- - 

•  The  successful  use  of  an  ISD  analysis  to  evaluate  a  training 
program  depends  upon  having  unbiased  subject  matter  experts; 
because  instructors  in  the  program  usually  provide  the  expertise 
there  is  inherent  bias  in  the  evaluation. 

•  The  lack  of  objective,  behavior- referenced  criteria  for 
assessing  team-oriented  skills  and  team  performance  limits 
the  STAN/EVA L  effort. 

•  Given  the  unvalidated  relationship  between  operations  training 
program  requirements,  in  terms  of,  for  example,  number  of 
intercepts  a  quarter,  and  skills  maintenance,  the  achievement 

of  those  requirements  should  not  be  used  as  an  evaluative  measure. 

•  .  The  measurement  tools  of  the  expert  are  unsophisticated,  informal, 

undocumented,  and  depend  on  the  experience  of  the  particular 
expert  involved. 
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•  Incomplete  documentation  of  ISD  analyses  complicates  program 
modification  and  makes  it  inefficient. 

•  Training  programs  are  occasionally  modified  as  a  result  of 
resource  shortages  or  changes  in  management  philosophy; 
such  modifications  frequently  reduce  the  quality  of  training. 

•  There  are  few  resources  for  and  very  low  priority  is  given  to 
modifying  simulators  which  are  known  or  demonstrated  to  be 
deficient  in  fidelity  or  capability. 

2 

A  salient  weakness  found  during  the  survey  was  a  lack  of  formal  T 
2 

programs  for  AFC  systems.  There  are  large-scale  exercises  for 

systems  and  joint  operations.  However,  they  have  tended  to  become 

ends  in  themselves  rather  than  being  based  on  and  targeted  to  the  job 

requirements  of  an  operational  environment  and  training  needs  or  objectives. 

2 

Exercises  are  limited  by  inadequate  simulation  capabilities  to  support  T 
and  the  unavailability  of  live  flying  resources. 

Current  measurement  techniques  do  not  support  evaluation  of  teams  or 
team  training  programs.  The  lack  of  diagnostic  measures  of  team  opera¬ 
tional  readiness  will  hamper  research  and  application  efforts  aimed  at 
increasing  it. 

A  critical  problem,  low  retention  of  experienced  individuals,  exists  in 
2 

the  C  career  fields.  The  contributing  factors  mentioned  by  the  survey 
respondents  were  in  the  areas  of  low  job  satisfaction  and  dissatisfaction 
with  career  opportunities.  Great  pressure  is  exerted  on  the  training 
pipeline  to  provide  proficient  replacements.  Given  the  importance  of 
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subject  matter  experts  throughout  the  training  cycle,  the  rapid  turnover 
and  loss  of  experienced  individuals  threatens  the  foundations  of  effective 
Air  Force  operations. 

The  low  priority  given  to  human  factors  and  training  in  system  acquisition 

exacerbates  the  problems  that  already  exist  and  causes  others.  Poor  , 

system  design  can  sometimes  be  overcome  through  training  but  this  is 

not  desirable  nor  does  it  make  effective  use  of  training  resources.  A 

limited  number  of  actual  equipment  trainers  are  available  and  they  have 

limited  capabilities  to  support  training,  especially  in  the  areas  of  training 

functions  to  support  student  and  instructors.  This  deficiency  is  a  result 

of  limited  resources  for  training,  unavailability  of  knowledge  to  address  * 

2 

T  requirements  to  procure  simulators  and  training  devices,  and  inadequate 
2 

provision  for  T  requirements  in  the  design  and  procurement  phases  for 

2  * 

C  systems. 


r 
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CHAPTER  II 


STATUS  OF  AFC2T2  PROGRAMS:  DEFINITION  AND  DEVELOPMENT 

OVERVIEW  AND  GENERAL  ISSUES 

Training  program  definition  and  development  involves  the  identification  of 

knowledge  and  skills  to  be  trained,  determination  of  the  level  of  proficiency 

to  be  achieved,  and  specification  of  how  the  training  is  to  be  accomplished 

and  evaluated.  A  number  of  factors  influence  the  overall  efficiency  and 

effectiveness  of  the  entire  definition  and  development  process.  General 
2 

factors  include  C  system  design,  joint  system  acquisition,  management 
philosophy,  and  fiscal  constraints.  These  general  factors  are  discussed 
in  this  section.  Specific  topics  are  discussed  in  the  following  sections. 
They  consist  of  assessment  of  training  needs,  training  requirements 
analysis,  transition  from  requirements  to  the  program,  and  simulator 
definition  and  development. 

2 

C  System  Design 

Training  is  not  a  panacea  for  poor  system  design  but  it  may  have  to 
overcome  the  effects  of  poor  design  to  achieve  adequate  proficiency. 

The  keyboard  of  the  AWACS  Situation  Information  Display  (SID)  console 
serves  as  a  ready  example. 
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An  alphabetic  keyboard  is  used  to  enter  call  signs  and  other  test  data. 

The  keys  are  sprung  quite  stiffly  to  avoid  inadvertent  entry;  further,  they 
are  arranged  in  an  unconventional  format: 


A  E  I  M  Q  U  Y 

B  F  J  N  R  V  Z 

C  G  K  O  S  W 

D  H  L  P  T  X 

A  set  of  special  characters  complements  the  keyboard.  Touch  typing 
is  difficult  and  slow  because  of  the  spring  tension  and  keyboard  layout. 

A  frequent  remark  was  that  a  typewriter  keyboard  layout  would  have  been 
more  sensible  from  a  training /human  factors  perspective. 

There  are  ample  opportunities  during  the  acquisition  cycle  to  incorporate 
good  human  factors  in  system  design.  Data  item  descriptions  require 
the  development  of  plans,  design  approaches,  and  data  bases  to  ensure 
that,  a  system  is  easily  usable  by  humans:  DI-H-7053  (Reference  11), 
DI-H-7056  (Reference  12),  and  DI-H-3268A /Q-118-2  (Reference  13). 

In  fact,  these  data  items  are  made  part  of  the  contractual  obligation  during 
system  acquisition.  But  because  system  acquisition  primarily  means 
the  procurement  of  hardware,  software,  and  their  capabilities,  the  human 
factor  often  waits  at  the  end  of  the  line  when  it  comes  to  contract  performance. 
According  to  interview  data,  the  reality  is  that  even  when  the  data  items  are 
required,  the  enforcement  of  the  requirements  is  incomplete  or  lax  as  a 
result  of  pressures  to  develop  the  system  itself. 
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The  reasons  for  this  low  priority  of  human  factors  are  not  clear  and  it 
is  debatable  whether  they  can  be  determined.  The  deliberations  that 
result  in  short  shrift  for  human  factors  occur  at  many  points  during 
budget  allocation  and  system  design.  They  are  typically  informal  rather 
than  the  outcome  of  a  systematic  cost-benefit  analysis.  Further,  they 
are  unrecorded  and  thus  not  retrievable  for  analysis  and  evaluation. 

Our  respondents  who  commented  on  this  issue  usually  attributed  it  to 
more  importance  being  placed  on  hardware  and  number  of  systems. 
However,  that  reasoning  is  circular.  The  reasons  must  lie  in  character¬ 
istics  of  human  factors  and  the  acquisition  process. 

Some  currently  popular  explanations  include  assertions  that  human  factors 

are  considered  too  late  in  system  design;  acquisition  cost  rather  than  life 

cycle  cost  is  the  principal  criterion  for  acquisition.  However,  other 

contributing  factors  might  be  the  lack  of  evidence  or  methodology  to  show 

during  development  that  human  factors  can  make  a  comparative,  cost- 

effective  contribution  to  system  effectiveness  and  the  absence  of  sufficient 

data  to  identify  and  support  critical  human  factors  decisions.  These 

factors  are  especially  true  for  the  cognitive  information  processing  semi- 

2 

structured  tasks  which  are  common  in  C  teams. 

The  human  factors  specialist  is  seldom  able  to  state  quantitatively  the 
impact  of  a  system  feature  on  individual  or  system  performance  or  the 
loss  from  not  incorporating  a  human  factors  feature.  Comparatively, 
the  engineering  disciplines  can  state  quantitatively  what  effects  their 
recommendations  will  have  on  parameters  such  as  range,  speed,  or  fire 
power.  Whatever  the  reason,  training  today  must  often  compensate  for 
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poor  design  features,  inadequate  man- machine  interfaces,  or  deficiencies 
in  system  capability.  This  expectation  is  unreasonable  and  has  the  con¬ 
sequences  of  Increased  training  costs  and  reduced  proficiency.  Of  course, 
the  exact  cost  and  reduction  involved  are  unknown. 

Joint  System  Acquisition 

2 

There  are  several  C  systems  in  acquisition,  for  example,  TRI-TAC 
and  BETA,  in  which  different  branches  of  the  military  are  procuring 
different  components  of  the  system.  For  example,  the  Army  has  overall 
responsibility  for  TRI-TAC,  but  the  Air  Force  is  procuring  the  Central 
Nodal  Communications  Element,  one  of  the  manned  components  of  the 
system.  The  Army  and  Air  Force  have  different  technical  career  fields, 
training  philosophies,  and  quality  of  personnel.  The  potential  problems  for 
the  training  developer- -Air  Training  Command  (ATC)--are  enormous 
under  these  circumstances.  Issues  of  compatibility  of  classifications  for 
personnel  and  skill  levels,  minimum  levels  of  proficiency,  and  compatibility 
of  training  objectives  must  be  resolved  before  training  can  be  developed. 

Management  Philosophy  and  Fiscal  Constraints 

The  combination  of  overall  Air  Force  mission  requirements  and  budget 

2  2 

cuts  or  restraint  puts  a  great  strain  on  AFC  T  training  definition  and 

development.  Airplanes  have  the  highest  priority;  in  competition  for 

2 

scarce  resources,  they  win  out  in  all  respects  over  C  systems.  Further¬ 
more,  a  general  hardware  orientation  has  a  negative  impact  on  training 
2 

within  AFC  .  This  is  true  for  systems  acquisition  and  ongoing  training. 
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For  example,  if  the  available  funds  will  buy  an  additional  system  capability 
or  a  training  device,  generally  the  system  capability  wins  out;  if  there 
is  a  choice  between  training  pilots  and  training  weapons  controllers,  pilots 
are  trained. 

This  is  not  intended  as  a  criticism,  although  there  are  some  circumstances 

in  which  decision  criteria  should  be  criticized.  Rather,  it  is  simply  to 

point  out  a  fact  of  life  regarding  Air  Force  training,  especially  in  an 

operational  setting.  Training  developers  must  do  the  best  they  can  with 

limited  resources  and  any  recommendations  for  improving  the  quality 
2 

of  AFC  training  must  take  this  fact  into  account. 

SPECIFIC  FACTORS 

In  addition  to  these  general  or  background  factors,  there  are  specific 
factors  affecting  the  efficiency  of  four  different  activities  of  the  training 
program  definition  and  development  process.  These  four  activities  are:  1 ) 
assessment  of  training  needs,  2)  training  requirements  analysis,  3) 
transition  from  requirements  to  the  training  program,  and  4)  simulator 
definition  and  development.  In  the  following  sections  each  of  these 
activities  is  described  in  terms  of  the  methods  employed  and  their 
products.  Strengths  and  weaknesses  of  the  activities  are  discussed. 

When  applicable,  (he  steps  of  the  generalized  ISD  model  presented  in 
Chapter  I  are  used  to  organize  the  description  of  each  activity. 
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Assessment  of  Training  Needs 


Training  program  definition  and  development  is  initiated  by  specific  needs. 

2 

Training  needs  in  AFC  arise  from  one  of  two  circumstances:  1)  system 
acquisition  or  modification,  or  2)  the  achievement  and  maintenance  of 
operational  readiness.  These  two  circumstances  require  different  needs 
assessment  techniques,  as  the  following  discussion  will  illustrate. 

System  Acquisition/Modification- -The  system  acquisition  process  is 
formal  and  carefully  regulated.  An  acquisition  cycle  begins  with 
definition  of  a  system  concept,  progresses  through  concept  validation  and 
engineering  development,  and  ends  with  production  and  deployment.  Thus 
an  accompanying  training  program  should  be  defined  and  developed 
simultaneously.  However,  this  is  rarely  the  case,  especially  for 
transition,  continuation,  and  exercise  programs.  Training  needs  are 
identified  during  concept  validation,  more  fully  defined  during  engineering 
development,  and  finalized  when  a  system  goes  into  production. 

The  system  acquisition  process  is  guided  by  a  Program  Management  Plan 
(PMP)  which  is  developed  by  the  staff  of  a  newly  organized  System 
Program  Office  (SPO).  Section  11  of  the  PMP,  a  living  document  that 
evolves  as  the  system  concept  is  fleshed  out,  deals  with  personnel  and 
training.  It  is  the  ATC  training  manager's  responsibility  to  ensure  that 
as  the  program  progresses  the  description  of  the  training  program  and 
related  topics  becomes  definitive.  Further,  the  training  manager  is  to 
develop  a  System  Training  Plan  (STP)  that  details  actions  required  to 
implement  ATC- controlled  training  for  a  new  system. 
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The  initial  step  in  assessing  the  training  need  during  concept  validation 
is  a  system  analysis.  This  is  a  joint  undertaking  by  ATC,  AFSC,  TAC, 
and  system  developer  representatives.  An  important  output  of  the  system 
analysis  is  the  personnel  requirements  of  the  system.  These  requirements 
provide  the  set  of  jobs  which  must  be  performed,  and  ultimately  trained, 
to  operate  and  maintain  the  system. 

It  was  not  within  the  scope  of  this  effort  to  determine  the  precise  nature 

of  the  methods  used  by  experienced  Air  Force  and  industry  personnel 

to  define  system  personnel  requirements.  However,  there  is  no  agreed- 

2 

upon  method  to  define  performance  requirements  in  C  systems  or  team 
training.  Thus  engineering  methods  influence  training  requirements 
and  determine  the  information  available. 

The  effectiveness  of  any  method  in  determining  system  personnel 
requirements  will  vary  with  the  nature  of  the  system  being  analyzed. 

The  relevance  of  the  new  system  to  existing  systems  is  a  key  factor 
influencing  the  validity  of  the  jobs  identified.  For  example,  the  personnel 
structure  of  JSS  overlays  that  of  the  SAGE.  Likewise,  the  jobs  to  be 
performed  within  JSS  are  highly  similar  to  those  performed  within  SAGE. 
Consequently,  the  establishment  of  a  valid  personnel  structure  is  relatively 
straightforward. 

On  the  other  hand,  the  establishment  of  the  AWACS  personnel  structure 

was  difficult  because  of  the  smaller  team  size  as  compared  to  SAGE  and 

TSQ-91.  This  team  size  change  altered  the  AWACS  personnel  job 

descriptions  compared  to  the  similar  positions  in  SAGE  and  TSQ-91 

2 

and  altered  the  familiar  team  structure  of  AFC  systems. 
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Thus,  the  system  designer  and  users  could  not  readily  draw  on  a  body 
of  knowledge  about  function,  skills,  and  training  objectives.  The 
personnel  subsystem  has  to  evolve  out  of  analysis  of  man- machine 
functions,  extrapolation  of  experience  and  trial  and  error.  Simulation 
and  system  analytic  techniques  may  some  day  provide  tools  to  design 
and  compare  different  concepts  for  personnel  configuration.  However, 
they  are  not  available  now  and  probably  would  be  too  costly  and  time- 
consuming  if  they  were. 

Regardless  of  the  specific  method  through  which  the  system  personnel 
structure  is  established,  the  identification  of  distinct  jobs 
initiates  training  development.  The  need  for  training  console  operators 
and  support  personnel  in  an  individual  performance  context  is  very  well 
accepted.  Though  the  job  positions  are  identified,  the  need  for  training 
supervisors  and  battle  staff  is  not  formally  established  during  system 
acquisition. 

2 

The  acquisition  of  computer-based  AFC  systems  establishes  a  need  for 

definition  and  development  of  simulation.  Recognition  of  this  need  is  based 

on  the  requirements  to  create  conditions  under  which  system  training 

for  developmental  test  and  initial  deployment  can  take  place.  The  radar 

2 

display  consoles  which  are  integral  to  AFC  systems  must  be  simulated 
in  order  to  provide  the  cues  for  operator  performance.  Live  flying  is 
not  a  realistic  possibility  for  accomplishing  system  training  during 
development  and  production.  Consequently,  computer  programs  which 
simulate  radar  returns  must  be  developed  for  use  on  the  actual  equipment. 
The  definition  and  development  of  simulation/simulator  requirements  is 
discussed  in  the  final  section  of  this  chapter. 


Modification  of  a  deployed  system  requires  assessing  training  needs  within 
the  context  of  ongoing  training  programs.  The  assessment  process  during 
modification  is  decentralized  as  compared  to  system  acquisition.  This 
decentralization  reduces  the  efficiency  of  the  need  assessment,  but 
because  training  program  modification  is  easier  than  definition  and 
development,  the  impact  is  not  great. 

Achievement  and  Maintenance  of  Operational  Readiness- -The  primary 

2 

peacetime  mission  of  AFC  organizations  is  training  in  order  to  achieve 
and  maintain  operational  readiness.  Under  these  circumstances,  the 
primary  responsibility  for  assessing  the  training  need  shifts  from  ATC 
to  TAC  personnel.  The  factors  influencing  the  efficiency  of  the  decision 
process  shift  also  as  a  consequence  of  TAC's  overall  mission  and  management 
structure.  The  decision  context  is  less  formal;  training  needs  are  based 
on  professional  judgment  of  readiness  to  meet  the  threat.  Individuals 
who  perceive  a  training  need  must  lobby  with  their  colleagues  and  superiors 
to  arrive  at  a  consensus. 

Over  the  years  the  need  to  have  operational  training  programs  consisting 
of  transition,  continuation,  and  system  training  has  been  firmly  established 
in  this  manner. 

Transition  training  meets  the  training  needs  of  console  operations  and 
support  personnel  and  this  informal  lobbying  is  effective  in  identifying 
training  needs  at  this  level.  They  are  effective  in  capturing  and  using 
the  knowledge  of  experienced  personnel  and  the  results  of  experience  with 
training  on  the  system.  This  approach  should  be  examined  to  delineate 
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more  clearly  how  it  is  done  and  how  it  works.  Perhaps  it  can  then  be 
adopted  for  use  in  training  requirements  analysis  for  continuation  and 
system  training. 

« 

Informal  lobbying  and  conferencing  characterizes  needs  assessment  and 
requirements  analysis  that  lead  to  definition  and  development  of  large- 

* 

scale  system  exercises  like  Blue  and  Red  Flag  and  training  courses 

taught  at  the  Air  Defense  Weapons  Center /interceptor  Weapons  School 

(ADWC/IWS).  They  seem  to  work  less  well  than  they  do  in  transition 

2 

training.  Most  recently  the  perception  of  a  lack  of  training  for  C  in  joint 

2 

service  operations  has  made  interoperability  a  key  AFC  training  need. 

I 

System  programs  and  large-scale  exercises  are  intended  to  address  team 
and  superteam  training  needs  and  provide  a  context  for  QJT  of  supervisors 
and  battle  staff.  They  need  improvement  which  should  start  with  the  * 

determination  of  training  requirements . 

At  least  once,  the  assessment  of  training  need  by  TAC  has  resulted  in  the 
definition  and  development  of  a  TAC  initial  training  program,  the 

Automatic  Positionally  Qualified  (APQ)  course  originally  developed  in  * 

1975.  Initial  training  is  for  the  most  part  the  province  of  ATC  rather  than 
TAC.  The  course  is  an  intermediate  step  between  the  ATC  fundamental 
manual  skill  qualification  course  and  initial  transition  training  at  the 
607  TCTS  or  a  SAGE  division. 
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It  is  not  clear  why  this  course  is  taught  by  TAC.  Certainly  TAC  established 
the  training  need  and  had  the  equipment  and  facility  available  within 
ADWC/IWS.  However,  ATC  should  have  the  course  responsibility  because 
it  is  not  oriented  toward  system- specific  skills.  The  existence  of  this 
situation  is  indicative  of  an  incomplete  Air  Force  policy  regarding  the 
relationship  between  training  need  assessment  on  the  one  hand  and  program 
definition  and  development  on  the  other. 

In  summary,  training  development  starts  with  the  assessment  of  training 
needs  for  specific  jobs  identified  in  a  system.  The  assessment  techniques 
needed  are  different  for  new  acquisition  and  maintenance  of  operational 
readiness  in  existing  systems.  There  are  differences  in  the  information 
available,  the  personnel  who  must  participate  in  the  development,  and  the 
agencies  which  are  responsible  for  training  development.  However,  existing 
procedures  for  determining  training  needs  are  designed  primarily  for  the 
situations  of  new  acquisitions  and  individual  training  managed  by  training 
command.  Training  for  achievement  and  maintenance  of  operational 
readiness  is  the  responsibility  of  the  operational  command;  existing 
procedures  are  inadequate  for  assessing  training  needs  in  that  context. 

TRAINING  REQUIREMENTS  ANALYSIS 

Once  a  training  need  has  been  established,  the  training  which  must  be 
developed  can  be  defined  through  a  training  requirements  analysis  (TRA). 
Following  the  generalized  ISD  model,  the  steps  in  TRA  involve  1)  identifying 
job  requirements,  2)  selecting  tasks  to  be  trained,  3)  analyzing  selected 
tasks,  and  4)  constructing  job  performance  measures.  Each  step  has  a 
product  associated  with  it.  For  example,  job  requirements  are  represented 
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as  a  prioritized  list  of  the  job  performance  tasks.  As  there  is  no  Air 
Force  regulation  that  they  be  formally  documented,  the  products  of  a  TRA 
(excepting  job  performance  measures)  are  rarely  found  unless  a  specific 
requirement  existed  at  the  time  it  was  performed;  for  example,  a  system 
developer  may  be  contractually  obligated  to  perform  a  job  task  analysis 
and  deliver  the  documented  results  to  the  government.  , 

An  exception  to  this  situation  was  found  at  the  4444th  Tactical  Air  Control 

System/Office  of  Training  Development  (TACS/OTD)  at  Luke  AFB.  The 

unit  personnel  have  documented  the  TRAs  produced  for  the  TSQ-91  Air 

Weapons  Controller  and  Air  Surveillance  Officer /Technician.  These 

products  are  indicative  of  the  efficiency  and  dedication  with  which  the  • 

TRAs  were  undertaken.  They  provide  a  basis  for  describing  and  evaluating 

the  TRA  techniques  that  are  employed  to  define  requirements  for  first- 

phase  transition  training  for  TSQ-91  console  operators  and  support  personnel. 

Because  ISD  methods  are  used  to  define  training  requirements  for  other 

initial  and  advanced  training  programs,  this  description  and  evaluation 

should  apply  to  those  methods  in  a  general  way,  except  where  the  survey 

discovered  evidence  to  the  contrary. 

2 

Definition  of  AFC  operations  training  requirements  in  taC  is  the 
responsibility  of  HQ  TAC/DOAO  for  AWACS  and  DOCTO  for  TACS. 

These  requirements  take  the  form  of  minimum  numbers  of  events  or 
hours  per  topic  or  activity.  They  are  not  the  result  of  a  formal  TRA. 

Expert  professional  judgment  serves  as  the  basis  for  determining  the 
requirements;  for  example,  in  establishing  die  mix  between  live  and 
simulated  flying.  In  the  case  of  live  events  or  activities,  the  requirements 
are  constrained  by  the  number  of  live  sorties  of  different  aircraft  types 
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that  are  projected  for  a  certain  peiiod  of  time,  for  example,  a  month. 

Thus,  the  training  developed  to  promote  achievement  and  maintenance 
of  operational  skills  is  not  defined  according  to  documented  methods  and  is 
not  based  solely  upon  training  requirements.  Both  of  these  facts  are 
characteristic  weaknesses  of  operations  training  programs. 

The  situation  described  above  also  characterizes  large-scale  exercise 
programs.  At  best,  the  training  requirements  established  are  constrained 
by  resource  considerations;  at  worst,  there  is  no  analytic  requirements 
definition. 

In  summary,  formal  TRAs  are  applied  only  to  individual  jobs  and  skills  in 
initial  training  programs.  Guidelines  do  not  exist  for  defining  individual 
and  team  training  requirements  for  operations  and  large-scale  exercise 
training.  This  definition  rests  on  professional  judgment  and  is  heavily 
influenced  by  non- training  factors. 

The  following  discussion  of  the  four  ISD  steps  in  TRA  describes  the  methods 
employed  and  the  factors  bearing  on  the  effectiveness  of  the  methods. 

The  steps  are  identification  of  job  requirements,  selection  of  training 
requirements,  analysis  of  tasks,  and  construction  of  job  performance 
measures. 

Identification  of  Job  Requirements 


This  step  has  been  called  the  "keystone  of  the  ISD  process"  (Reference  10). 
The  purpose  of  this  step  is  to  produce  a  valid  list  of  job  tasks  prioritized 
in  terms  of  frequency,  difficulty,  criticality,  and  the  like.  These  tasks  are 
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then  used  as  the  basis  for  all  subsequent  requirements  analyses.  The 
adequacy  of  the  procedures  for  this  step  for  describing  jobs  of  teams  in 
command  and  cost  role  is  critical. 

Existing  procedures  are  established  by  the  requirements  of  DI-H-6130 
(Reference  14),  Tasks  and  Skill  Analysis  Report.  Job  tasks  are  defined 
as  part  of  system  development  for  acquisition  of  new  systems.  Depending 
upon  the  relevance  of  the  new  system  to  existing  systems,  there  may  be 
task  lists  available  from  existing  occupational  surveys  or  the  Military 
Task  Data  Rank.  Even  if  these  data  are  not  specifically  available,  the  job 
requirements  for  new  system  operation  are  usually  related  to  existing 
specialties.  Typically,  the  system  developed  can  depend  upon  resident 
military  job  experts  to  identify  job  requirements.  The  resulting  list  of 
job  tasks  is  documented  as  part  of  a  Task  and  Skill  Analysis  Report 
(DI-H-6130),  a  data  item  which  the  government  requires  on  most  system 
development  / production  contracts . 

DI-H-6130  includes  only  general  guidelines  for  identifying  job  requirements. 

Further,  the  scope  is  limited  to  equipment  operators.  Failure  to  adequately 

address  unique  jobs  or  tasks,  non-console  tasks,  and  tasks  of  non-operators 
2  2 

results.  Many  AFC  T  jobs  consists  of  these  types  of  tasks.  In  addition, 

DI-H-6130  does  not  provide  task  priorities.  It  needs  to  be  augumented  to 
2  2 

meet  the  needs  of  C  T  . 
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When  a  new  system  departs  significantly  in  technology  or  operation  from 
existing  systems,  unique  jobs  or  job  tasks  may  be  created.  If  this  occurs, 
job  experts  either  do  not  exist  or  they  are  hampered  in  identifying  tasks 
because  of  unknown  or  incomplete  operating  procedures.  In  such  situations 
the  training  requirements  will  be  incomplete  at  best  and  invalid  at  worst. 

Summarizing  the  deficiencies  of  DI-H-6130  applied  to  new  systems,  it  does 
not  include  guidance  regarding  different  possible  types  of  tasks  and  their 
relation  to  mission  objectives,  although  the  identification  of  job  tasks  is 
required  to  be  related  to  the  total  mission.  Individual  tasks  and  tasks 
requiring  team  efforts  are  not  distinguished  from  one  another.  Consequently, 
training  requirements  for  team  skills  are  not  explicitly  defined.  The  tasks 
of  non- operators,  for  example,  certain  support  personnel,  supervisors, 
and  battle  staff,  are  not  specifically  identified.  If  training  is  to  be  developed 
for  these  personnel,  then  the  burden  of  TRA  falls  on  ATC.  Finally,  in 
DI-H-6130,  there  is  no  requirement  to  provide  task  priorities  in  terms 
of  frequency,  criticality,  or  difficulty.  These  data  have  an  impact  on 
selection  of  tasks  for  training. 

One  consequence  of  the  first  and  third  shortcomings  noted  above  is  clearly 
evident  in  AWACS  training.  Although  the  Senior  Director  (SD)  and  Mission 
Crew  Commander  (MCC)  have  jobs  involving  some  console  operation,  no 
job  task  lists  have  been  developed  because  these  are  supervisory  positions. 
So,  despite  the  fact  that  the  SD  and  MCC  job  include  some  operator  tasks, 
there  is  no  formal  training  of  them. 


Where  initial  and  advanced  training  is  defined  and  developed  by  TAC,  expert 
judgment  is  again  employed  to  derive  task  lists.  The  job  requirements  are 
more  clearly  related  to  operational  mission  requirements.  Take,  for 

■ 

example,  the  task  listing  produced  by  TACS/OTD  for  the  TSQ-91  Weapons 

Controller.  It  is  very  general,  as  shown  in  Table  7.  The  eight  tasks  are 

basic  to  the  job  of  a  weapons  controller.  Five  of  the  eight  tasks  (2  through  % 

6)  imply  interaction  with  other  team  members,  and  thus  they  may  be 

considered  the  forerunner  of  team  training  requirements.  The  remaining 

three  tasks  refer  to  equipment  operating  procedures.  The  task  priority 

data  consist  of  an  index  of  the  importance  (impact)  of  correct  task  performance 

and  the  task  learning  difficulty.  This  information  is  potentially  important 

in  making  tradeoffs  in  training  program  development.  » 

In  developing  the  list  of  tasks  for  Weapons  Controller  performance,  the 
general  conditions  of  a  combat  theater  are  used.  If  garrison  conditions 
are  used,  some  lesser  tasks  would  have  to  be  added.  On  the  other  hand, 
several  very  significant  tasks  would  have  to  be  eliminated.  Table  7  presents 
a  general  listing  of  tasks  performed  by  the  CRC/CRP  positional  Weapons 
Controller.  Included  with  the  listing  is  a  rank  ordering  from  one  to  three, 
an  indication  of  the  importance  of  correct  task  performance,  and  the 
learning  difficulty  of  the  task  performance. 

In  summary,  the  guidelines  for  identifying  job  requirements  during  system 
acquisition  are  incomplete.  They  do  not  relate  well  to  the  purpose  of  the 
TRA  and  provide  no  guidance  for  distinguishing  between  individual  and  team 
tasks.  Application  of  the  ISD  guidelines  for  identifying  job  requirements 
produces  task  lists  consistent  with  the  goals  of  the  TRA.  However,  here, 
too,  there  is  no  explicit  recognition  of  team  skills. 
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TABLE  7.  TSQ-91  WEAPONS  CONTROLLER  TASK 


Number 

Task 

Impact 

Learning 

Difficulty 

Task  1 

Prepares  radar  and  radio  con¬ 
sole  to  provide  control  ser¬ 
vices  to  aircraft. 

1 

Easy 

Task  2 

Initiate  or  accept  handover/ 
handoff  of  aircraft. 

3 

Easy 

Task  3 

Provides  flight  advisory  infor¬ 
mation  (as  necessary  or 
requested)  to  aircraft  en  route 
to/from  mission  area. 

2 

Moderate 

Task  4 

Provides  positioning  assistance 
and  target/rendezvous /objective 
briefing  to  aircraft  for  speci¬ 
fic  aircraft  mission  requirements. 

i 

Difficult 

Task  5 

Provides  enemy  threat  informa¬ 
tion  to  aircraft  in  mission 
area  (the  area  of  Aerial  Combat 
Tactics). 

i 

Moderate 

(Complex) 

Task  6 

Records  and  passes  mission 
results/data  to  other  agencies. 

3 

Easy 

Task  7 

Implements  radar  console  and 
communications  ECCM  procedures. 

2 

Easy  (Complex) 

Task  3 

Shuts  down  radar  and  radio  con¬ 
soles. 

3 

Easy 

Provides  control  services  to  aircraft  in  a  declared  emergency 
situation. 

Originally  identified  as  Task  8,  however. 

determined  is 

only  a  change  in  "condition"  while  task  performance 
remains  same  as  in  Tasks  2,  3,  and  4. 
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Selection  of  Tasks  for  Training 


According  to  ISD,  some  tasks  should  be  selected  and  others  rejected  for 

training  based  upon  the  priorities  of  the  task  list,  training  resources,  and  * 

cost.  The  survey  produced  no  evidence  of  selectivity  of  tasks  per  se. 

Rather,  all  the  tasks  identified  as  job  requirements  are  trained  to  one 

I 

degree  or  another,  or  under  some  conditions  and  not  others,  with  no 
apparent  rationale.  It  is  possible  that  this  is  an  artifact  of  the  listing 
process  in  that  only  those  tasks  to  be  trained  are  listed.  Level  of  task 
proficiency  to  be  achieved  and  conditions  of  task  performance  are  determined 
later  in  the  definition  and  development  process. 

For  system  and  large-scale  exercises,  the  system  tasks  which  are  trained 

are  determined  by  the  particular  resources  available.  For  example,  if  an 

AWACS  is  available  for  a  live  TACS  exercise,  then  the  tasks  associated  with  * 

TACS-AWACS  coordination  will  be  trained.  If  an  AWACS  is  not  available, 

no  training  in  such  tasks  occurs.  The  number  of  live  intercept  exercises 

is  determined  by  the  number  of  fighter  aircraft  scheduled  to  fly  rather  than 

the  training  needs  of  personnel  in  surveillance  and  weapons  control. 

n 

Analysis  of  Tasks 

The  purpose  of  task  analysis  is  to  determine  under  what  conditions  a  task 
must  be  performed  and  what  standards  of  performance  are  required  to  do 
the  job.  Task  analysis  results  in  a  detailed  description  of  tasks  and  task 
steps,  and  associated  environmental  and  equipment  conditions  and  behavioral 
criteria  of  performance. 
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Task  analysis  is  a  difficult,  time-consuming  procedure  which  must  be 
carried  out  by  skilled  professionals.  Both  behavioral  and  subject  matter 
expertise  must  be  available.  During  system  development  a  great  deal  of 
imagination  is  necessary  to  describe  tasks  that  will  be  performed  on 
equipment  that  does  not  yet  exist  under  conditions  that  are  not  completely 
specifiable.  Because  production  time  may  be  short  or  the  required  expertise 
unavailable,  as  when  the  system  contractor  must  meet  a  demanding  schedule 
within  cost,  the  results  of  task  analysis  are  rarely  complete,  detailed 
descriptions  of  job  tasks  and  performance  conditions  or  criteria. 

The  instructions  for  preparation  of  the  Task  and  Skill  Analysis  Report  are 
,  very  clear  regarding  what  information  to  include.  However,  there  are  two 

additional  factors  which  influence  the  quality  of  the  report.  First,  there 
are  few  guidelines  on  the  methods  to  be  used  in  obtaining  the  information. 
Second,  task  analysis  is  a  low-priority,  low-budget  activity  relative  to  the 
overall  system  development/production  process. 

These  factors  result  in  inconsistent  methods  being  applied  by  individuals 
who  vary  in  level  of  expertise  and  knowledge.  A  high  degree  of  imagination 

•  and  creativity  is  needed  to  describe  job  steps  in  the  absence  of  a  physical 
system.  Time  and  budget  constraints  work  against  this  need.  Finally, 
the  amount  of  effort  which  is  put  into  task  analysis  will  be  the  minimum 

* 

necessary  to  satisfy  the  letter,  not  the  intent,  of  the  data  item  requirement. 
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The  task  analysis  obtained  from  the  4444th  TACS/OTD  does  not  contain 
all  of  the  information  required  by  either  ITRO  or  the  Air  Force  ISD  model 
(see  Table  8).  For  example,  performance  conditions  are  described  only 
briefly  and  the  performance  standards  are  not  specified.  Included  for 
each  task  element  are  the  type  of  performance  required  (discrimination, 
recall,  problem-solving,  manipulation,  or  speech)  and  the  learning 
difficulty  (easy,  moderate,  difficult,  or  very  difficult).  These  data  are 
useful  for  making  decisions  related  to  training  development,  but  are 
not  directly  related  to  a  TRA. 

The  tasks  described  in  the  TACS/OTD  analysis  include  the  individual 

and  team-oriented  steps  and  elements  required  to  do  the  job.  A  statement 

of  training  requirements  is  derived  by  comparing  this  description  against 

an  assessment  of  entry-level  skills  (a  separate  ISD  step  discussed  under 

transition  from  requirements  to  program).  The  training  requirements  » 

(see  Table  9)  are  presented  in  such  a  general  way  that  the  specificity  of  the 

task  analysis  is  lost.  Consequently,  in  the  analysis  there  is  no  definitive 

statement  of  the  individual-  or  team-oriented  skills  to  be  trained.  This 

state  of  affairs  is  a  function  of  the  format  and  method  chosen  for  the 

analysis  rather  than  a  result  of  factors  related  to  the  nature  of  the  tasks  * 

or  the  training  program. 

Construction  of  Job  Performance  Measures 


Interviews  and  observations  did  not  reveal  the  methods  used  in  constructing 
job  performance  measures.  The  end  result  of  the  process  is  a  standard 
form  consisting  of  factors  to  be  evaluated  and  instructions  regarding  the 
scoring  technique. 
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TABLE  8.  TAC/OTD  WEAPONS  CONTROLLER  TASK  ANALYSIS 


Task  3:  Provides  flight  advisory  information  (as  necessary  or 
requested)  to  aircraft  en  route  to/from  mission  area  (for  example, 
weather,  traffic  advisories,  friendly  artillery). 


Step  Steps  in  Performing  Principal  Type  Learning 

Number  the  Task  of  Performance  Difficulty 

3.1  Recognize  when  to  provide  Discrimination  Moderate 

the  various  types  of  in¬ 
formation. 

3.2  Recognize  specific  in-  Discrimination  Moderate 

formation  that  should  be 

provided. 

3.3  Locate  information.  Recall  Easy 

3.4  Read  information.  Manipulation /recall  Moderate 

3.5  Report  that  information  Recall/speech  Moderate 

to  aircraft  in  appropriate 

format  and  sequence. 


Skills  and  Knowledge--Task  3:  Know  radio  R /T.  Read 
weather  display.  Read  artillery  display.  Operate  radio 
subset.  Operate  telephone  subset.  Discriminate  friendly 
radar  tracks  that  pose  "stranger  threat"  to  each  other. 


TABLE  8.  TAC/OTD  WEAPONS  CONTROLLER  TASK 
ANALYSIS  (concluded) 


Task  4:  Provide  positioning  assistance  and  target/rendezvous  briefing 
to  aircrew  for  specific  aircraft  mission  requirements. 


Step 

Number 

Steps  in  Performing 
the  Task 

Principal  Type 
of  Performance 

Learning 

Difficulty 

4a 

(Subtask)  Select /confirm 

tactic  and  technique  to  be 
employed. 

4a.  1 

Note  relative  location  of 
aircraft  and  the  desig¬ 
nated  target /rendezvous / 
objective. 

Discrimination 

Easy 

4a.  2 

Determine  control  aircraft 
capabilities  and  mission 
requirements. 

Discrimination/ 

recall 

Very  diffi¬ 
cult 

4a.  3 

Determine  target/rendez¬ 
vous /objective  parameters 
or  capabilities  (move¬ 
ment.  formation,  etc). 

Discrimination/ 

recall 

Very  diffi¬ 
cult 

4a.  4 

Evaluate  "conditions  "  for 
mission  impact. 

Discrimination 

Difficult 

4a.  5 

Recall  alternative  tactics 
and  techniques. 

Recall 

Difficult 

4a.  6 

Select  appropriate  tactic 
and  technique  to  be 
employed. 

Problem  solving/ 
discrimination 

Very  diffi¬ 
cult 

Skills  and  Knowledge--Task  4(a):  Read  Offensive  Mission 
Display.  Know  basic  capabilities  of  primary  friendly  tacti¬ 
cal  aircraft  (speeds,  arms,  manuverability).  Know  basic 
capabilities  of  expected  enemy  aircraft  (speeds,  arms,  manu¬ 
verability).  Read  frag  data.  Read  track  ARO. 


TABLE  9.  TSQ-91  WEAPONS  CONTROLLER  TRAINING  REQUIREMENTS 


Task  3;  Provide  Flight  Advisory  Information- -Student  was  required  in 
ATC  course  *o  provide  "stranger  threat"  information  and  weather  information. 
Training  requires  proficiency  practice  in  those,  plus  any  new  in  the  TACS 
(for  example,  friendly  artillery  warnings,  to  include  reading  the  display 
boards). 


Task  4:  Provide  Positioning  Assistance  and  Target  Briefing-- 
Subtasks 

a.  Select/confirm  tactics 

b.  Provide  guidance  positioning 

c.  Brief  aircrew  on  target  parameters 

d.  Coordinate  with  other  agencies 

e.  Release  aircraft  to  mission  final  control  agency 

In  the  ATC  school,  student  is  required  to  demonstrate  guidance  positioning 
on  airborne  target  interception  and  on  air  refueling  rendezvous.  It  is 
likely  the  tactic  (or  approach  to  target)  was  specified  in  advance.  Also 
was  required  to  brief  aircrew  using  "standard"  R/T. 

• 

Training  requirement  must  allow  student  to  acquire  ability  to  select 
the  best  approach  to  target  (tactic)  for  any  situation,  both  air-to-air 
and  air-to-surface  missions.  (Positioning  after  tactic  selection  is  a 
matter  of  gaining  proficiency  on-scope  and  learning  rules  of  thumb.) 

Associated  with  tactics  selection  is  knowledge  training  in  specific 
aircraft  capabilities,  aircrew  tactics  and  mission  objectives,  and 
flight  safety  considerations  that  are  pertinent  to  radar  control  efforts. 

1  For  the  most  part,  the  student  is  only  familiar  with  the  subtasks,  other 

than  select  tactic,  and  proficiency  is  expected  to  be  very  low.  Emphasis 
will  be  on  training  activities  that  allow  skill  proficiency.  Since  tactical 
engagement  parameters  for  interception,  etc,  are  short-range,  there 
*  should  be  no  need  for  proficiency  (that  is,  training)  with  the  CPU- 73 

Attack  Computer.  Rather,  knowledge  proficiency  training  should  be 
oriented  to  the  rules  of  thumb  in  solving  the  air  mass  problem,  with 
reference  to  the  attack  computer  only  as  an  additional  resource.  Since 
manual  skills  form  the  basis  of  AWC  abilities,  manual  skill  proficiency 
should  be  emphasized  with  the  HM4118  computer  utilization  skills  as 
additive. 
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The  factors  to  be  evaluated  are  directly  related  to  job  tasks.  Because 
different  positions  may  involve  the  same  or  closely- related  tasks,  one 
form  can  apply  to  several  positions.  For  example.  Air  Defense  Command 

« 

(ADCOM)  Form  745,  presented  in  Table  10,  covers  evaluation  of  Senior 

Directors /Technicians  and  Senior  Weapons  Directors /Technicians  for 

SAGE,  GYK-19,  and  manual  systems.  , 

Scoring  is  defined  in  terms  of  an  ordinal  rating  scale.  The  scale  may 

have  two  (satisfactory/unsatisfactory)  or  more  categories.  Several 

categories  are  described  in  terms  of  the  level  of  proficiency  or  knowledge 

which  must  be  demonstrated  in  order  for  the  performance  to  receive  the 

associated  score.  For  example,  on  a  0  to  5  scale,  0  indicates  no  knowledge  » 

of  the  task,  3  indicates  an  ability  to  do  or  knowledge  of  most  parts  of 

the  task,  and  5  indicates  knowledge  of  or  ability  to  do  the  complete  task 

quickly  and  accurately,  and  ability  to  tell  or  show  others  how  to  do  the 

task. 

2 

The  job  measures  used  throughout  the  C  systems  surveyed  are  of  this 
general  type.  The  specific  standards  of  performance  for  each  task  or 
evaluation  factor  are  not  explicitly  stated  on  the  form.  These  standards, 
if  they  are  documented,  sure  described  In  positional  handbooks  which  are 
discussed  in  the  next  section.  Often  the  standards  are  not  documented, 
but  rather  are  established  by  the  level  of  proficiency  or  knowledge  of  the 
most  experienced  job  experts.  This  makes  the  construction  of  job 
performance  measures  extremely  difficult  for  new  systems  and  has  a 
negative  impact  on  training  implementation  and  evaluation. 
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TABLE  10.  ADCOM  FORM  745 


SENIOR  DIRECTOR/TECHNICIAN  (SD/T)  AND 

SENIOR  WEAPONS  DIRECTOR/TECHNICIAN  (SWO/T)  EVALUATION  FACTORS 

(SAGE  GYK-19/Hmrmml  Syatmnm) 

9A  rt  camouc^zo 

*****  OP  >NO<viOUAi  EvaljATCO  i'Lmi.  lift,  middla  Mint) 

SPAOt 

ORCAHI JAT10W 

njpposc  op  cvalua?>o* 

- - - - 

SYSTEM  •RpUcaEi*  bow) 

C  SAOE  GYK.lt  Q2  MANUAL 

INSTRUCTIONS 

This  form  is  designed  co  provide  guidance  for  the  evaluation  of  the  position  specified  above.  Each  icen 
will  be  scored  0  through  5. 

0  -  Demonstrated  no  knowledge  of  the  task. 

1  -  Demonstrated  some  knowledge  of  the  task.  Needs  to  be  shown  or  told  how  to  do  the  task. 

2  -  Demonstrated  ability  to  do  or  knowledge  of  simple  parts  of  the  task. 

3  •  Demonstrated  ability  to  do  or  knowledge  of  most  parts  of  the  task. 

4  «  Demonstrated  ability  to  do  or  knowledge  of  all  parts  of  the  task.  Meets  standards  for  speed  and 

accuracy. 

3  *  Demonstrated  knowledge  of  or  ability  to  do  the  complete  cask  quickly  and  accurately.  Can  tell 
or  show  others  how  to  do  the  task. 

A  score  of  3  VI LL  be  attained  on  any  item  preceded  with  a  m. 

Any  score  below  4  VI LL  be  explained  on  reverse  of  form,  prefixed  with  the  appropriate  item  number. 
Veak  or  recommended  improvement  areas  VI LL  be  noted  on  reverse  of  form,  also  prefixed  with  the  appro¬ 
priate  item  number. 


knowledge  op  on  fhoficjency  in 
evaluation  factors 

— 

SCORE 

Continumtton 

*CORE 

1.  MISSION  »L  AN  NINO  'Include  mtm »  brltto*) 

SUBTOTAL  BROUGHT  FORWARD 

2.  LRM.'AKWaC  OPERATIONS  ANO  RAOAR 
CAPABILITIES 

■  14.  ARTCC  PROCEDURES,  MAJOR  AIRWAYS.  ANO 
RESTRICTED  AREAS  i BSCAT/SCATARA) 

■  J.  LEVELS  OP  OPERATION  ANO  TRANSITION 
PROCEDURES 

IS.  WEAPONS  CAPABILITIES  (Jnfwd—t—  mnd  ADA) 

0  A.  COMPUTER  PROGR  AM/STA  RTOVCR/SWI TCMOVER 
PROCEDURES 

IS.  INITIAL  STOP  MANNINO  ANO  PIONTER  PLOW 

S.  NAOtO  ANO  TOOL  CAPABILITIES  ANO 
PROCEDURES 

B  IT.  WEAPONS  EMPLOYMENT /RULES  OP  SN9A0S- 
MENT /TRACK  ASCIONMENT 

■  1.  INTERCEPTOR  SOUAOPONS.  SASCS.  ANO 

TAPOCr  COMPLEXES 

IS.  MONITORINO  WEAPONS  AVAILABILITY  ANO 
WEATH8R  CONDITIONS 

IS.  LOCAL  POLICIES  ANO  PROCEDURES 

a  s.  PLC  PNOCf  OUPCS/ ALIPT  NOT:  Pi  CA  TION  ANO 
POPMAT  OlSSEMI  NATION  PROCEDURES 

20,  CONSOLE  UTILIZATION  ANO  KNOWLEOOS  OP 

displays 

CONTROL  CASES 

21.  USE  OP  AVAILABLE  aids  fCh+ektlmf.  imm 
>Mr<«.  »te.> 

10.  CMCROENCT  ACTIONS  ANO  REPORTlMO 
PROCEDURES 

B  22.  INTERNAL  ANO  EXTERNAL  COORDINATION 

INCLUDING  NANOL1NO  OP  INTELLiOCNCE  INPO 

n.  *ACTICAL  ACTION  RCPOPTINO  PROCEDURES 

B  U.  SUPERVISION  ANO  KNOWLEOOS  OP 

SUBORDINATE  SECTIONS 

IS.  NO-NOTICE  EXERCISE  PROCEDURES 

24.  PTe  PROORAM  ANO  PROCEDURES 

■  IS.  UNKNOWN  TRACK  PROCEDURES 

2S.  R'T  ANO  COMMUNICATIONS  DISCIPLINE 

subtotal 

TOTAL 

rows 

AOCOM  .in  "  745 


MtV'OUl  COITION  is  OIIOLtrt. 
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The  survey  did  not  discover  any  analytic  or  other  methods  being  used  to 
construct  team  or  system  performance  measures.  No  documented 
measures  of  team  readiness  exist.  The  only  existing  measure  of  team  or 
system  effectiveness  is  percentage  of  targets  detected  and  intercepted 
during  a  live  or  simulated  exercise.  The  characteristics  of  this  measure¬ 
ment  technique  are  described  in  Chapter  III. 

Thus,  there  is  a  lack  of  explicitly  specified  methods  for  developing 
performance  measures,  scores,  and  empirical,  demonstrated  relationship 
between  performance  measures  and  the  readiness  of  a  team  or  unit. 

The  consequence  of  these  tasks  is  uncertainty  concerning  the  validity  and 
utility  of  the  performance  measures  for  indicating  proficiency  and  training 
needs,  if  they  have  any  validity  at  all.  These  conditions  also  foster 
practices  of  measuring  what  can  be  measured  readily  in  order  to  satisfy 
a  bookkeeping  requirement,  rather  than  developing  measures  to  gather 
information  on  what  ought  to  be  known  to  evaluate  performance  and 
training  adequately. 

TRANSITION  FROM  TRAINING  REQUIREMENTS  TO  TRAINING  PROGRAM 

The  generalized  ISD  model  described  earlier  includes  12  steps  to  be 
carried  out  in  the  transition  from  training  requirements  to  training  program. 
In  order  io  more  closely  approximate  the  manner  in  which  this  transition 
is  accomplished  in  the  Air  Force,  these  12  steps  are  collapsed  into  six, 
as  follows: 

•  Select  setting  and  type  of  training 

•  Develop  training  objectives 
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•  Determine  and  develop  proficiency  assessment  techniques 

•  Select  instructional  methods,  media,  and  sequencing 

•  Develop  plan  for  program  management  and  materials 
production 

•  Produce /validate  instruction 

In  the  Air  Force  these  six  steps  (and  thus  this  portion  of  the  ISD  process) 
are  accomplished  with  reference  to  regulations  specifying  how  and  when 
certain  training- related  documents  must  be  prepared.  These  documents 
include  course  training  standards  (CTSs)  required  by  Air  Force  Regulation 
8-13,  and  curriculum  documentation  such  as  course  charts,  plans  of 
instruction  (POIs-- equivalent  to  syllabi  developed  by  TAC),  and  lesson 
plans  required  by  ATC  Regulation  52-6.  During  system  acquisition,  CTSs 
and  course  charts  become  part  of  a  system  training  plan  (STP)  used  by  ATC 
to  guide  instructional  development.  POIs,  syllabi,  lesson  plans,  and 
other  necessary  training  material  are  produced  in  accordance  with  the  STP. 

These  documents  provide  a  ready  means  for  recording  decisions  made 
during  the  training  program  development  process.  In  fact,  to  a  large 
degree,  the  ISD  requirement  in  the  Air  Force  is  met  by  producing  these 
documents  (Reference  10).  This  may  be  more  a  function  of  the  unit  or 
organization  carrying  out  the  ISD  process,  as  evidenced  by  documentation 
of  training  requirements  analyses  found  at  the  4444th  TACS/OTD. 
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In  the  following  discussion  of  the  six  development  steps,  the  purpose  and 
contents  of  the  various  documents  are  described  in  conjunction  with  the 
pertinent  step.  The  documents  reveal  the  extent  and  character  of 
development  decisions  that  are  made.  They  also  provide  some  insight 
into  methods  used  and  their  strengths  and  weaknesses. 

Select  Setting  and  Type  of  Training 

2 

The  assignment  of  AFC  training  to  a  particular  location  or  situation  is  in 
large  part  dictated  by  tradition.  In  this  there  is  both  a  strength  and  a 
weakness.  Tradition  tends  to  be  based  upon  what  works,  a  strength. 

On  the  other  hand,  tradition  tends  to  be  inflexible  and  does  not  provide 
for  consideration  of  new,  perhaps  better,  alternatives,  a  weakness. 

2 

Initial  and  advanced  training  of  C  operators  and  support  personnel  takes 
place  in  resident  schools  and  TAC  resident  training  units.  Transition 
and  continuation  training  and  system  exercises  take  place  in  operational 
units. 

Initial,  advanced,  and  some  transition  training  are  characterized  by 
formal  courses  whose  development  is  discussed  in  this  section.  The 
remaining  transition,  all  continuation  and  system  training  are  primarily  on- 
the-job.  On-the-job  training  (OJT)  is  informal,  involving  individual 
participation  in  a  certain  number  of  intercepts  or  activities  per  month 
or  quarter.  System  training  involves  unit  participation  in  a  certain  number 
of  exercises  per  month  or  quarter. 


Transition  and  continuation  training  for  supervisors  and  battle  staff  is 
carried  out  on-the-job.  OJT  is  a  default  training  setting  for  these 
personnel  because  no  initial  training  needs  are  established.  It  is 
traditional  to  advance  skilled  operators  and  support  personnel  into 
supervisory /staff  positions.  For  example,  weapons  directors  in  a 
CRC  come  from  the  ranks  of  the  weapons  controllers.  The  OJT  setting 
for  training  supervisors  and  staff  works  reasonably  well  for  systems 
which  have  been  deployed  for  some  time  because  there  is  a  pool  of 
experienced  individuals  available.  However,  during  system  acquisition 
the  tradition  is  not  optimal  because  there  is  no  experienced  pool  of 
operators,  support,  or  supervisory  personnel  from  which  to  advance 
individuals. 

Development  of  initial  training  of  operators  and  support  personnel  for 
new  or  modified  systems  is  managed  by  ATC.  There  are  four  types  of 
initial  training  development  which  ATC  manages.  Type  I  is  developed 
and  conducted  by  a  contractor,  usually  the  system  developer /producer. 

The  setting  for  Type  I  training  may  be  the  factory  or  the  field,  depending 
on  the  need  for  or  availability  of  (training)  equipment.  The  purpose  is  to 
provide  a  trained  crew  to  man  the  system  during  development  and  operational 
tests  and  to  train  instructors  who  will  carry  out  Type  II  and  Type  IV 
training.  Type  II  training  is  carried  out  by  ATC  using  contractor-developed 
draft  materials.  The  purpose  of  this  training  is  to  meet  the  initial  manpower 
demands  of  the  new  system.  Thus  Type  II  training  is  characterized  by 
high  throughput.  Type  III  training  takes  place  in  resident  schools.  It 
is  based  on  revised  and  validated  course  materials.  The  purpose  of  Type 
III  training  is  to  meet  personnel  replacement  needs  in  the  system  life 
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cycle  context.  Type  IV  training  is  conducted  at  operational  sites  by  ATC 
mobile  training  teams.  The  purpose  of  Type  IV  training  is  to  provide 
a  cost-effective  means  of  accomplishing  training  in  the  field  when  dictated 
by  (training)  equipment  availability. 

Some  examples  will  elarify  the  relationship  between  setting  and  type  of 
training.  All  existing  ATC  courses  for  personnel  are  Type  III.  AWACS 
training  is  a  combination  of  Type  III,  formal  ADWC/IWS,  formal  TAC 
initial  training,  and  QJT.  In  the  case  of  JSS,  Type  I  and  Type  IV  training 
will  be  used  to  satisfy  system  testing  and  some  early  site  manning 
requirements.  Follow-on  training  needs  will  be  satisfied  by  Type  IV, 
formal  ADWC/IWS,  and  QJT. 

System  training  is  carried  out  under  live  or  simulated  conditions  within 
operational  units.  Although  conducted  within  the  context  of  formal  exercises, 
system  training  is  primarily  QJT  and  informal. 

Develop  Training  Objectives 

Whereas  training  requirements  analyses  deal  with  the  job,  the  development 
of  training  objectives  represents  a  shift  in  focus  to  the  specification  of  what 
skills  and  knowledge  are  to  be  trained.  Objectives  are  statements  of 
performance  requirements,  conditions,  and  standards.  Enabling  objectives 
may  be  distinguished  from  terminal  objectives. 
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Inherent  in  the  decisions  made  regarding  objectives  is  a  determination 
of  the  entry-level  skill  and  knowledge  of  the  students.  This  determination 
is  relatively  straightforward  for  all  courses  and  training  beyond  the 
fundamental  ATC  courses.  However,  entry-level  skill  requirements  for 
assignment  to  the  weapons  control  career  field  have  not  been  established. 
There  are  no  minimum  scores  on  the  Air  Force  Officers  Qualifying  Test 
(AFOQT)  or  other  capabilities  or  prerequisites  which  must  be  attained  or 
demonstrated  by  entrants  into  the  ATC  Air  Weapons  Controllers  Fundamentals 
Course.  Enlisted  entrants  into  the  Aerospace  Control  and  Warning  Systems 
Operator  career  field  must  achieve  a  score  of  at  least  60  on  the  general 
portion  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) . 

In  the  Air  Force,  training  objectives  are  documented  in  a  CTS.  The  CTSs 

for  the  ADWC/IWS  APQ  Air  Weapons  Controller  Course  and  the  JSS 

Weapons  Controller  course  are  presented  in  Appendix  C.  The  AFQ  CTS 

is  quite  detailed;  assumed  student  entry-level  skills /knowledges  are 

those  acquired  during  the  Air  Weapons  Controller  Fundamentals  Course. 

Team-oriented  training  objectives  are  implicit  rather  than  explicit.  The 

JSS  CTS  is  general;  assumed  student  entry-level  skills /know ledges  are 

those  acquired  during  SAGE  duty.  Training  for  JSS  is  thus  crosstraining 

from  one  system  to  another.  The  overall  objective  of  crew  training  has 

2 

not  been  analyzed  in  any  further  detail.  There  are  thus  no  formal  T 
objectives  for  the  JSS  program. 

The  training  objectives  for  AWACS  Weapons  Director  initial  transition 
training  have  been  documented  separately  in  greater  detail  than  that 
found  in  the  CTS.  Table  11  presents  a  representative  set  of  objectives 
from  this  document.  For  each  topic  related  to  knowledge  or  skill,  the 

71 


m 


TABLE  11.  EXAMPLES  OF  WEAPONS  DIRECTOR  COURSE 
(E-3A  00  COO  DX) 


OBJECTIVES 

1.  Tactical  Air  Control  System  Radar  Units 

(WEA:TCR)  Be  able  to  recall  the  name,  acronym,  and  basic  function 
of  each  of  the  TACS  radar  elements  without  error. 

2 .  Non-Radar  Air  Support  Coordination  and  Control  Elements 

(WEA.'TCN)  Be  able  to  recall  the  name,  acronym,  and  basic  functions 
of  each  of  the  TACS  Air  Support  Coordination  and  Control 
elements  and  other  non-radar  TACS  control  elements. 
Also,  recall  their  relative  relationships  in  the  system. 

3.  Inside  the  E-3A 

(WEA:E-3A)  Be  able  to  identify  crew  members  by  job  titles  when 
given  their  function. 

Be  able  to  match  the  appropriate  functional  group  with 
its  major  components. 

4.  Weapons  Section 

(WEArWEA)  Be  able  to  identify  Weapons  Section  tasks  and  name  the 
positions  that  perform  the  tasks. 

5.  Situation  Display  Console  (SDC)  Components 

(WEA:SDC)  Name  each  of  the  major  Situation  Display  Console 

(SDC)  components  and  associate  the  components  with 
their  functions. 

6.  Category  Select  Switches 

(WEA:CAT)  Associate  the  Category  Select  Switches  with  the 
displays  they  control. 

7.  SDC  Checkout  and  Setup 

(WEAiSET)  Check  out  and  set  up  a  Situation  Display  Console 

(SDC)  including  assigning  the  console  to  the  proper 
function. 

8.  Switch  Action  Rules 

(WEArRUL)  Identify  components  of  Input  and  Menu  Lines,  and  .... 
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EXAMPLES  OF  WEAPONS  DIRECTOR  COURSE  (concluded) 


(PTP:SPT)  Given  a  List  of  special  point  symbols  and  codes. 

create /display  a  special  point  type  at  any  coordinate 
or  location,  specifying  a  designator  or  DLRN, 
heading,  altitude,  and  speed  (if  appropriate). 

Delete  a  special  point  from  the  system. 

41.  Commit  to  Special  Point 

(PTP:CSP)  Use  computer  actions  to  conduct  three  simultaneous 
radar  monitor  and/or  control  missions  to  a  special 
point  target  using  specific  approach  parameters. 

The  aircraft  will  take  off  at  three-minute  intervals 
after  the  special  points  have  been  entered  into  the 
computer. 

Verify  the  information  contained  in  the  data  base  for 
assigned  point-to-point  missions. 

42.  Point-to-Point  Missions 

(PTP:MSN)  Recall  the  basic  functions  of  each  of  the  point-to-point 
missions  and  their  components. 

43.  Conduct  Point-to-Point  Missions 

(PTF:CPM)  Given  adequate  mission  briefings,  conduct  the 
following  point-to-point  missions: 

•  Close-air  support  with  FFAC  and  FAC-A  support 

•  Close-air  support  with  FFAC  support 

•  Close-air  support  with  ASRT  support 

•  Interdiction 

•  Interdiction  with  escort 

•  Reconnaissance 

•  Airlift 

•  Psychological  warfare 

•  Search  and  rescue 

44.  Point-to-Point  Handover  Procedures 

(PTP:HAN)  Conduct  five  simultaneous  (individually  controlled) 
point-to-point  missions,  separated  as  briefed,  but 
not  less  than  1000  ft  (2000  ft  at  or  above  .... 
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specific  actions  and  conditions  and  some  standards  are  described.  As 
above,  team-oriented  training  objectives  are  implicit  rather  than  explicit. 

(See  objectives  43  and  44  in  Table  11). 

The  CTS  may  be  viewed  as  a  contract  between  the  unit  doing  the  training 
and  the  unit  to  which  the  individual  will  report  next.  The  acquiring  unit 

« 

assumes  that  the  required  proficiency  levels  have  been  obtained.  These 

levels  become  the  basis  for  establishing  training  standards  in  the  acquiring 

unit  or  in  setting  the  criteria  for  accomplishing  transition  training.  In  the 

absence  of  a  CTS,  for  example,  in  GJT,  training  objectives  are  not 

specified  as  such.  Terminal  objectives  are  specified  as  part  of  the 

standardization /evaluation  testing  program.  Enabling  objectives  if  they  , 

are  specified  at  all  are  in  terms  of  the  number  of  live  or  simulated  events 

and  exercises  which  individuals  and  teams  must  participate  in  during  a 

month  or  quarter.  * 

There  are  no  formal  analytic  techniques  in  use  for  determining  individual 
objectives.  Job  experts  and  training  developers  specify  objectives,  including 
standards,  through  a  discuss  ion /negotiation  process.  The  results  of  the 
process,  except  standards,  are  documented  in  a  CTS.  •* 

The  same  sort  of  informal  process  is  used  to  develop  system  exercise 

t 

objectives.  The  appropriateness  and  precision  of  the  system  training 
objectives  depend  on  several  factors.  Because  the  process  is  informal 
and  dependent  on  the  participating  individuals'  experience,  training 
objectives  per  se  may  not  be  addressed  for  system  exercises  at  all.  If 
the  group  decides  to  develop  training  objectives,  they  may  be  constrained 
by  available  resources.  For  example,  if  it  were  desirable  to  make  a  live 
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exercise  extremely  difficult,  this  may  not  be  possible  given  the  number  of 
aircraft  available  or  the  types  of  jamming  which  can  be  employed  in 
peacetime  training. 

In  summary,  the  development  of  training  objectives  for  initial  and 
advanced  training  courses  is  documented  in  the  form  of  a  CTS.  The 
process  of  development  is  based  on  job  expertise.  Training  objectives 
are  not  developed  for  transition  and  continuation  training.  The  training 
requirements  regarding  participation  in  a  minimum  number  of  live  or 
simulated  intercept  events  or  exercises  are  not  translated  into  objectives. 
The  objectives  for  system  exercises  are  developed  informally,  if  at  all, 
and  are  usually  limited  by  non- training  considerations. 

Entry-level  skills  possessed  by  personnel  depend  upon  previous  courses 
and  job  experience.  There  are  no  minimum  requirements  or  prerequisites 
for  entry  into  the  weapons  control  career  field.  In  system  acquisition, 
training  objectives  for  Type  I  and  IV  courses  are  based  on  the  previous 
job  experience  of  the  personnel  to  be  trained.  Thus,  the  levels  of 
knowledge  and  skills  which  the  students  bring  to  the  course,  the  starting 
point  for  training,  is  known  with  uncertainty  and  is  probably  heterogeneous. 

Determine  and  Develop  Proficiency  Assessment  Techniques 


Proficiency  assessment  is  necessary  in  order  to  gauge  trainees'  progress, 
maintain  quality  control  over  course  graduates,  and  provide  a  basis  for 
upgrade  in  operational  status.  It  entails  the  specification  of  relevant  task 
dimensions  and  performance  characteristics,  and  establishment  of 
evaluative  criteria.  Feedback  to  trainees  is  inherent  in  the  proficiency 
assessment  process,  but  may  be  augmented  by  independent  techniques. 
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The  Air  Force  distinguishes  between  knowledge  and  skill  components  of 
proficiency.  Knowledge  concerns  the  facts,  principles,  and  theories  of 
the  subject  matter  to  be  mastered.  Skill  refers  to  the  actual  tasks  and 
duties  which  must  be  performed. 

The  measurement  of  performance  is  governed  by  the  dichotomy  of  job 
proficiency  components.  Knowledge  is  measured  using  objective  tests, 
usually  of  the  multiple  choice  variety.  In  the  case  of  duty-specific 
knowledge,  oral  exams  may  be  used.  Criteria  for  knowledge  objectives 
are  absolute  rather  than  relative.  For  example,  90%  on  a  test  is  passing, 
or  an  oral  exam  is  satisfactory  or  unsatisfactory.  Skill  is  measured 
by  observation:  experts  evaluate  positional  performance.  The  format 
of  the  evaluation  during  training  is  precisely  the  same  as  the  job  performance 
measures  discussed  previously.  The  criteria  for  speed  and  accuracy  of 
performance  are  not  well  documented;  they  are  internalized  by  the  expert 
based  on  job  experience. 

2 

In  C  there  is  a  very  close  relationship  between  training  objectives  and 
\ 

standards  as  defined  in  a  CTS,  on  the  one  hand,  and  the  performance 
factors  to  be  evaluated  and  criteria  on  the  other.  In  the  case  of  skills 
assessment,  the  translation  from  objectives  to  evaluation  factors  is 
straightforward,  but  tends  to  promulgate  any  shortcomings  of  the 
specified  training  objectives  into  proficiency  assessment. 
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For  example,  team-oriented  skills  are  evaluated  implicitly  rather  than 
explicitly.  Also,  where  no  training  objectives  have  been  developed,  as  for 
supervisors  and  battle  staff,  performance  measurement  dimensions  are 
s  based  solely  upon  job /duty  descriptions.  These  descriptions  tend  either 

to  be  quite  global,  for  example,  "supervision  and  teamwork  and  knowledge 
of  local  procedures"  (see  Table  10  for  additional  examples),  or  incomplete. 

The  survey  discovered  no  evidence  of  efforts  to  develop  proficiency  assessment 
techniques  specifically  designed  to  evaluate  team  or  system  readiness  or 
effectiveness.  As  discussed  in  the  next  chapter,  in  practice,  expert 
judgment  based  on  observation  of  team  performance  is  the  basis  of  team 
proficiency  assessment;  and  system  performance  is  measured,  if  at  all, 
by  the  percentage  of  successful  detections  or  intercepts /kills,  or  by 
expert  judgment  as  to  whether  or  not  the  mission  was  successful. 

In  summary,  the  determination  and  development  of  proficiency  assessment 
techniques  are  adequate  for  individual  performance  of  operators  and 
support  personnel,  but  have  actual  or  potential  shortcomings  in  the  team 
skills  area  and  for  supervisors  and  battle  staff.  These  shortcomings 
«  include  the  lack  of  explicit  objectives  and  standards  for  team  skills;  use 

of  global  job/duties  statement  in  lieu  of  training  objectives,  especially  for 
supervisors  and  battle  staff  positions;  the  lack  of  assessment  of  team  or 
system  readiness;  and  a  lack  of  documented  standards.  Feedback 
techniques  are  correlated  with  the  methods  of  training  and  of  proficiency 
assessment.  Individuals  receive  feedback  intrinsic  to  the  tasks  they 
perform  and  from  the  evaluator  based  on  observations  of  performance. 

Teams  receive  feedback  intrinsic  to  the  mission  they  perform,  including 
a  debriefing  after  it  has  been  completed.  More  will  be  said  about  feedback 
in  Chapter  III. 
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Select  Instructional  Methods,  Media,  and  Sequencing 

The  performance  dichotomy  between  knowledge  and  skill  training  objectives 

4 

traditionally  dictates  the  selection  of  academic  training  and  positional 
or  hands-on  training  activities,  respectively.  Selection  of  instructional 
methods  requires  decisions  about  the  degree  of  individualization,  type  of  , 

instructional  pacing,  student /instructor  ratio,  and  the  like.  Media  selection 
involves  specifying  the  precise  means  by  which  students  will  encounter 
instructional  material,  for  example,  lecture,  film,  simulation,  etc. 

Instructional  sequencing  is  selected  according  to  relationships  among  the 

training  objectives  including  commonality  and  dependence;  enabling 

objectives  must  be  met  before  terminal  objectives.  » 

2 

Methods  of  instruction  in  TAC  are  not  selected  by  analytic  means,  but 

tend  to  be  a  function  of  course  management  structure  or  philosophy.  For 

example,  the  ATC  initial  training  courses  are  group  lock-step  or  paced 

while  the  TAC  initial,  transition,  and  continuation  training  courses  are 

self-paced.  In  ATC  the  class  is  viewed  as  the  instructional  unit:  a  class 

enters,  is  instructed,  and  graduates.  In  TAC  the  instructional  unit  is 

more  often  the  individual  and  so  the  self-pacing  method  dominates.  Also, 

the  student- instructor  ratio,  which  is  maintained  at  between  2  and  3  to  1 
2 

throughout  C  training  programs,  is  a  function  of  the  mas  ter- apprentice  ( 

model  which  dominates  positional  training.  More  will  be  said  about  this 
model  in  Chapter  IV. 
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Although  there  are  analytic  techniques  for  determining  media  requirements, 
the  survey  found  no  evidence  of  their  use.  The  media  of  academic  training 
are  consistent  with  instructional  methods.  Lectures  augmented  by  written 
or  sound/slide  programs  are  universal  in  group  and  individual  instruction. 
Interestingly,  the  same  type  of  modularized  programmed  texts  and  sound/ 
slide  are  used  regardless  of  the  type  of  pacing.  This  appears  to  be  a 
byproduct  of  the  application  of  criterion-referenced  ISD  to  the  development 
of  academic  training,  and  a  tendency  on  the  part  of  some  training  developers 
to  equate  the  application  of  sound/slide  programs  with  satisfying  the  ISD 
requirement. 

The  need  for  simulation  as  a  positional  training  medium  is  a  given  for 
2 

C  operators  and  teams.  This  situation  appears  to  have  arisen  out  of  the 
need  to  exercise  teams  and  evaluate  system  performance  under  simulated 
conditions.  Live  flying  for  such  purposes  poses  problems  of  safety  and 
limited  resources,  for  example,  planes  and  fuel.  There  is  no  analytic 
technique  for  determining  the  relative  amounts  of  simulation  and  live  flying 
optimal  for  training.  Simulation  on  actual  equipment  has  become  a  primary 
medium  for  accomplishing  initial,  transition,  and  continuation  positional 
training  and  for  conducting  team/system  exercises.  The  definition  and 
development  of  simulation  training  are  characterized  in  this  chapter's 
final  section. 

Instructional  sequencing  is  governed  by  commonality,  dependence,  and 
complexity  for  individual  knowledge  and  skill.  Team-oriented  tasks  are 
not  subject  to  sequencing  because  they  sure  not  explicitly  identified  as 
training  objectives. 
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In  summary,  the  selection  of  instructional  methods,  media,  and  sequencing 
is  more  a  function  of  non- training  factors  like  tradition,  management 
philosophy,  and  constraints,  or  shortage  of  aircraft  and  fuel,  than  it  is  a 
function  of  training  considerations.  From  the  point  of  view  of  team  ‘ 

readiness,  the  most  critical  aspect  of  this  state  of  affairs  is  the  degree 
to  which  simulation  can  provide  adequate  training  and  experience. 

I 

Develop  Plan  for  Program  Management  and  Materials  Production 

The  survey  discovered  that  only  during  system  acquisition  was  a 

comprehensive  instructional  management  and  production  plan,  the 

System  Training  Plan  (STP),  developed.  The  STP  contains  guidelines  f 

and  actions  required  to  implement  A  TO  controlled  training;  it  is  an 

outgrowth  of  Section  11  of  the  PMP  as  discussed  earlier.  There  is  no 

comparable  single  document  developed  for  managing  and  producing  • 

transition  and  continuation  training.  Rather,  the  elements  of  such  a  plan 

are  addressed  through  manuals,  regulations,  and  separate  CTSs  and 

course  descriptions. 

The  key  features  of  a  program  management  and  materials  production  plan  * 

should  be:  1)  guidelines  for  the  preparation  of  lesson  content,  2)  description 

of  procedures  for  student  management,  3)  discussion  of  the  selection, 

training,  and  roles  of  instructors,  and  4)  administrative  information 

including  class  sizes,  schedules,  and  training  equipment  needs.  In  short, 

the  plan  should  document  all  relevant  decisions  made  to  this  point  in  the 

training  development  process,  plus  additional  information  to  smooth  the 

transition  to  the  training  program.  The  STP  and  the  operations  training 

planning  documents  lack  certain  of  these  features.  A  description  of  both 

will  highlight  their  strengths  and  weaknesses. 
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The  STP  Includes  the  following  information,  which  is  updated  at  six-month 
intervals  during  system  development  (Reference  15).  It  identifies  the 
organizations  involved  in  implementation  and/or  support  of  training  and 
their  responsibilities  including  authoring;  it  describes  types  of  training 
required  over  the  system  life  cycle,  including  plan  for  developing  organic 
training  support;  it  provides  personnel  data  including  numbers  and  types 
of  students  and  instructors  required  and  when;  it  describes  the  training 
facilities  required  to  support  training;  it  describes  the  required  training 
program  including  courses,  location,  type,  length,  and  start  data,  plus 
for  each  identified  job,  the  training  requirements;  it  describes  the  CTS 
and  course  chart  (a  description  of  the  topics  to  be  covered,  their  sequence 
and  the  length  of  time  spent  on  each);  it  identifies  the  technical  data 
needed  to  support  training;  and  finally,  it  provides  an  overall  training 
schedule  based  on  system  test  and  evaluation  needs. 

The  STP  does  not  specifically  address  student  management  procedures 
because  ATC  has  standard  procedures  applicable  to  all  training.  The 
STP  does  not  discuss  selection  of  instructors  or  students  beyond  specifying 
entry-level  skills  for  students.  As  noted  earlier,  the  CTS  (and  the  course 
chart)  deal  with  team  training  requirements  and  objectives  at  a  global 
level,  if  at  all.  Finally,  training  equipment  and  its  capabilities  are  not 
discussed  in  the  STP.  These  issues  are  normally  addressed  under  the 
system  development  contract,  as  described  in  the  next  section. 
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TAC  does  not  develop  such  unified  plans  dealing  with  the  management  of 
transition,  continuation,  and  system  training.  The  most  representative 
sources  would  be  TAC  Manual  50-9  (Reference  5)  and  TAC  Regulation 
51-43,  which  deal  with  criteria  and  procedures  to  conduct  the  operations 
training  program  for  TACS  and  AWACS,  respectively.  The  operations 
training  program  consists  of  transition  and  continuation  training  for 

« 

individuals,  and  simulated  and  live  system  exercises. 

In  these  documents,  individual  requirements  are  stated  in  terms  of 

minimum  hours  for  subject  area  coverage  (academic)  and  minimum  number 

of  hours  of  types  of  events  or  activities  (positional)  for  all  TACS  personnel. 

Conditions  under  which  these  requirements  must  be  met  and  may  be  , 

modified  are  also  given.  Training  objectives  and  other  more  detailed 

information  are  not  included.  Lesson  outlines  and  plans  are  published 

by  TAC  HQ  operations  training  offices.  Because  each  operational  unit  * 

implements  these  outlines  and  plans  and  each  unit  has  local  procedures 

to  follow,  the  specific  content  of  local  operations  training  programs  will 

vary,  especially  in  the  area  of  positional  training  and  system  exercises. 

However,  TAC  HQ  publishes  information  guides  for  self-study  and  these 

guides  are  based  on  the  TACS  Stan/Eval  Program;  so  training  of  the  f 

knowledge  component  of  proficiency  is  standardized.  The  4444th  TACS/OTD 

ISD  team  is  referenced  in  TAC  Manual  50-9  and  it  is  stated  that  their  efforts 

i 

will  periodically  result  in  new  requirements  and  materials  for  training 
TACS  operators.  These  replace  existing  requirements  and  materials 
as  they  are  fielded. 
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Management  of  personnel  in  training  status  is  thoroughly  discussed. 
Transition  training  is  divided  Into  two  phases,  in  which  the  purpose  of 
Phase  I  is  familiarization  and  Phase  n  is  aimed  at  achieving  operational 
ready  status.  Continuation  training  is  accomplished  in  Phase  m.  Phase  IV 
is  a  non-training  status  reserved  for  Unit  Commanders,  Chiefs  of 
Operations  and  Plans,  NCOICs,  and  other  high-level  supervisors;  all  of 
these  personnel  pass  directly  from  completion  of  Phase  I  to  Phase  IV.  Their 
day-to-day  activities  are  expected  to  maintain  their  operational  readiness. 

Personnel  requirements  for  instructors  are  identified  in  terms  of  highly 
qualified  individuals  who  have  demonstrated  (he  ability  to  instruct  and  are 
completely  'knowledgable  in  all  relevant  subject  areas  (Reference  5).  This 
approach  seemed  generally  to  be  the  case  in  practice,  although  highly 
qualified  and  completely  knowledgable  people  are  becoming  harder  to  find. 
There  is  also  competition  for  these  people  in  operational  positions. 

System  exercises  are  carried  out  under  simulated  or  live  conditions. 
Provision  is  made  in  TACS  for  different  categories  of  simulated  exercises 
and  minimum  training  requirements  in  terms  of  frequency  are  established. 

No  specific  requirements  or  plans  are  given  for  live  system  exercises. 

The  categories  of  simulated  exercises  depend  on  the  scope  of  involvement 
of  TACS  elements.  Category  1  is  training  which  exercises  a  partial  TACS 
varying  from  a  single  to  several  elements  (for  example,  a  CRC  and 
ASRT).  Category  2  is  a  system  netted  training  activity  designed  to  fully 
exercise  interactions  among  all  elements  of  the  TACS.  Category  3  is 
training  which  exercises  the  TACC  Current  Plans  Division  (PLANEX). 

The  minimum  training  requirement  for  each  category  is  determined  by 
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expert  judgment  and  tradition  which  follows  the  SAGE  model.  Participation 
in  live  exercises  fulfills  simulated  exercise  requirements  on  a  slightly 
better  than  one-for-one  basis. 

Guidance  for  participation  in  training  in  a  live  environment  notes  that  such 
opportunities  usually  arise  in  joint  service  exercises.  It  is  the  responsibility 
of  TACS  units  to  coordinate  with  other  services'  c2  systems  to  provide 
live  training  opportunities.  Such  coordination  is  encouraged  because 
establishing  the  necessary  data  links  makes  the  experience  extremely 
beneficial  since  these  links  would  be  critical  in  a  theatre  operation. 

We  discovered  no  comparable  documents  for  the  planning  of  large-scale 
exercises  although  schedules  are  published  yearly.  Such  planning  is 
usually  accomplished  in  conferences  which  are  attended  by  representatives 
of  participating  elements  and  commands.  These  experts  determine  the 
scope  and  objectives  of  the  exercise  consistent  with  its  general  purpose. 

The  exact  details  are  based  upon  available  equipment  and  personnel 
resources  and  safety  considerations.  A  scenario  to  drive  the  exercises 
is  devised  through  decisions  based  on  professional  judgment.  Some 
series  of  exercises  are  wholly  simulated,  while  others  combine  simulated 
(scripted)  and  live  (flying)  inputs.  The  definition  and  development  of 
radar  and  event-based  simulated  inputs  are  discussed  in  this  chapter's 
final  section. 
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Produce /Validate  Instruction 


The  final  steps  in  training  program  definition  and  development  are  the 
production  and  validation  of  instruction.  Instructional  materials  are 
produced  for  academic  and  positional  training.  New  materials  are 
evaluated  and  revised  as  necessary  before  actual  use.  These  steps 
are  accomplished  in  essentially  the  same  manner  for  all  training  programs 
surveyed. 

2 

When  a  C  system  is  produced,  if  it  is  computer-based,  then  the  government 
can  require  the  developer  to  provide  positional  handbooks — DI-M-3409/H-109-1 
(Reference  16).  According  to  the  data  item  description,  "positional  handbooks 
describe  the  operating  procedures  for  consoles  used  to  support  the  system 
in  accomplishing  its  prime  mission.  "  The  emphasis  of  each  handbook  is  on 
the  responsibilities,  duties,  and  operating  procedures  required  of  the 
operational  position.  The  handbook  is  capable  of  serving  as  a  primary 
instructional  tool  for  training  operators  of  new  systems.  The  survey  did 
not  determine  the  extent  to  which  this  data  item  is  or  has  been  required  on 
C2  system  programs. 

For  initial  and  advanced  training  programs,  instructional  and/or  subject 
matter  experts  produce  instructional  materials  with  the  assistance  of 
audio  visual  aid  experts  if  necessary.  These  materials  take  the  form 
of  plans  of  instruction,  lesson  plans,  syllabi,  programmed  texts,  and  the 
like.  The  content  of  these  materials  follows  from  CTSs,  course  charts, 
and  more  detailed  sources  if  they  exist.  These  documents  are  universally 
well-structured,  clear,  and  complete  insofar  as  previous  analyses  have 
been  thorough.  Lesson  plans  and  syllabi  specify  the  subject  matter  topics 


m- 
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and  objectives,  and  conditions  under  which  demonstration  and  practice 
of  positional  skills  take  place.  Their  purpose  is  to  ensure  that  all 
students  are  exposed  to  a  minimum  standard  set  of  problem  types  and 
procedures. 

In  operations  training  programs,  materials  beyond  those  provided  by  TAC 

* 

HQ  may  or  may  not  be  produced  at  the  discretion  of  the  unit  operations 
training  officer.  SAGE  Regional  Control  Centers  each  produce  their 
own  materials  within  the  framework  of  the  A  DCOM- established  requirements 
and  objectives.  Operational  TACS  elements  and  AWACS  squadrons  which 
have  intensive  initial  transition  training  programs  work  with  the  HQ-provided 
materials. 

Instructional  materials,  per  se,  are  not  produced  to  support  system  exercise 
programs  or  large-scale  exercises.  Briefing  materials  are  written  by  exercise  • 
planners.  The  briefings  establish  conditions  and  objectives  in  the  form 
of  mission  requirements,  thus  initiating  the  exercise  scenario. 

Under  ideal  circumstances  all  instructional  material  produced  as  part 

of  ISD  is  subject  to  a  tryout,  a  simulation,  with  a  sample  of  individuals  * 

representative  of  the  student  population.  Rarely  does  the  Air  Force  have 

the  luxury  of  evaluating  newly-produced  materials  in  this  manner.  Initial 

validation  of  materials  is  by  approval  of  HQ  SMEs.  Evaluation  and 

modification  of  instructional  materials  result  primarily  from  use  in 

training. 
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TRAINING  DEVICE  DEFINITION  AND  DEVELOPMENT 


This  section  summarizes  our  major  observations  regarding  the  definition 

2 

and  development  of  AFC  training  simulators  and  simulation  exercises 
that  were  observed  during  the  site  visits.  The  principal  simulations 
observed  were: 

i 

•  Manual  radar  sets  (UPA  35  and  62) 

•  SAGE 

•  CRC/CRP/TACC 

•  BUIC 

•  AWACS 

Basic  weapons  controller  training  (Tyndall)  is  supported  by  manual  radar 
consoles  that  are  stimulated  with  prerecorded  simulated  radar  returns 
(T-2  tapes)  and  interactively  controlled  simulated  tracks  (T-4  tracks). 
The  T-4  tracks  are  controlled  by  enlisted  personnel  who  play  the  role  of 
interceptor  pilots  (T-4  drivers).  Up  to  four  T-4  tracks  can  be  controlled 
by  one  T-4  driver,  and  12  tracks  can  appear  on  one  display.  T-4  tracks 

i 

can  be  overlaid  on  top  of  T-2  or  live  radar  imagery.  The  student  console 
is  manned  by  the  student  controller  and  another  student  who  is  playing  the 
«  role  of  weapons  technician.  The  instructor  and  students  are  in  two-way 

communication  with  the  T-4  driver  who  mimics  the  verbal  responses  of  a 
pilot  who  has  received  directions  from  the  controller. 
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The  SAGE  simulation  exercises  (Luke  AFB,  Duluth)  were  conducted  in 

2 

operational  SAGE  centers  and  exercised  the  entire  C  team.  The 

scenarios  involved  attacks  on  the  continental  US  by  manned  bombers,  and 
2 

the  role  of  the  C  team  was  to  detect  the  attackers  and  direct  interceptors 
against  them.  The  scenarios  were  presented  in  the  form  of  prerecorded 
tapes  that  stimulated  operational  SAGE  equipment. 

\ 

The  CRC/CRP/TACC  exercises  were  multisite  events  that  involved  a 

CRC  (Eglin),  TACC  (Shaw),  and  a  number  of  ancillary  units.  Events  on 

operational  CRC  scopes  were  portrayed  by  T-2  recordings  and  T-4  tracks. 

Communication  links  were  established  between  participating  sites.  Some 

2 

participants  in  the  C  network  were  played  by  script  readers  who  provided 
prepared  verbal  inputs  at  specified  times  within  the  exercise. 

The  BUIC  simulator  (Tyndall)  is  used  in  an  AQP  course  that  is  designed  • 

to  transition  student  weapons  controllers  from  manual  to  automated  radar 
systems.  The  simulator  consists  of  operational  BUIC  equipment  stimulated 
by  systems  that  are  functionally  comparable  to  the  T-2  and  T-4  systems. 

The  AWACS  simulator  (Tinker)  includes  nine  operational  mission  crew  ♦ 

consoles  (situation  displays)  and  a  computer  display  maintenance  operator 
display  (CDMO)  in  a  room  that  resembles  the  interior  of  the  E3A  aircraft, 
as  well  as  interceptor  pilot  simulator  (IPS)  consoles  (similar  in  function  f 

to  T-4  driver  stations)  in  an  adjacent  room.  Simulation  exercises  use 
prerecorded  (T-2  style)  tapes  in  conjunction  with  interactive  (T-4  style) 
tracks  to  stimulate  operational  equipment. 
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In  all  systems  the  instructors  monitored,  assisted,  and  evaluated  student 
performance  by  standing  behind  the  students  and  watching  the  display  and 
the  operator's  actions. 

In  all  cases  the  physical  fidelity  of  the  simulators  was  excellent  because 
operational  equipment  was  used.  Serious  problems  were  observed  in  the 
utilization  of  the  simulators  for  training,  however,  and  these  problems 
are  discussed  in  detail  in  the  remainder  of  this  section.  It  is  our  opinion 
that  most  of  the  problems  have  resulted  from  the  failure  of  the  simulator 
definition  and  development  process  to  coordinate  with  an  expanded  ISD 
process.  Therefore,  our  discussion  is  organized  around  four  major  steps 
that  may  be  abstracted  from  the  ISD  framework: 

•  Functional  analysis 

•  Partitioning  of  functions 

•  Definition  of  training  device  requirements 

•  Development  of  a  training  device 

By  organizing  the  discussion  in  this  way  we  intend  to  underscore  the  major 
steps  which  in  our  opinion  should  be  followed  in  the  training  device  definition 
and  development  process.  Most  of  the  simulations  cited  above  were 
developed  before  ISD  processes  were  formalized.  Even  the  AWACS 
simulator,  which  is  the  newest  one  we  observed,  was  initially  designed 
early  in  the  history  of  the  ISD  model  although  the  most  recent  copy  of  the 
simulator  was  delivered  in  1980.  Thus,  it  is  not  surprising  that  many 
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of  the  guidelines  discussed  herein  were  not  followed.  Our  purpose  is  not 

to  demonstrate  the  quality  of  hindsight,  but  to  document  lessons  learned 
2 

so  that  future  AFC  simulators  will  provide  higher- quality,  more  cost- 
effective  training. 

Functional  Analysis 


Functional  analysis  is  related  to  the  job  or  task  analysis  produced  in  ISD. 

The  goal  is  to  identify  and  define  the  functions  performed  by  the  operator 

2 

in  the  course  of  his  job.  Examples  of  operator  functions  in  AFC  systems 
include  interpreting  radar  imagery,  selecting  an  appropriate  display  scale 
and  center,  performing  range  and  bearing  computations,  and  guiding  an 
intercept.  A  functional  analysis  identifies  and  defines  the  stimulus  conditions 
and  information  the  operator  must  act  on  and  the  procedures  he  must  follow. 
The  analysis  should  be  broader  than  the  details  of  the  software  and  hardware 
of  the  system,  and  should  include  a  consideration  of  the  tactical  environment 
in  which  the  system  will  operate.  Examples  of  relevant  factors  are  the 
density  and  capabilities  of  own  and  threat  forces,  combat  tactics,  and 
tactical  mission. 

We  have  no  direct  data  bearing  on  the  types  of  functional  analysis  that  were 
performed,  if  any,  in  the  design  of  the  simulators  we  observed,  but  the 
reports  of  interview  respondents  indicated  that  the  analyses  were  either 
not  performed  or,  at  most,  were  conceived  narrowly.  The  set  of  simulator 
functions  was,  in  general,  apparently  defined  in  terms  of  the  actual  system. 
This  is  particularly  understandable  because  all  systems  utilize  operational 
equipment  stimulated  by  simulated  radar  returns.  The  problem  with  this 
approach  is  that  operational  variables  affecting  operator  functions  were 
sometimes  overlooked. 
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An  example  of  this  problem  is  apparent  in  the  T-2  system,  which  presents 
prerecorded  radar  imagery.  Because  the  imagery  is  prerecorded  it  is 
noninteractive.  The  operator  cannot  talk  to  simulated  pilots  and  observe 
responses  on  the  scope.  Thus,  although  the  scope  and  function  keys 
function  realistically  in  the  simulation,  the  operator's  communication 
functions  are  not  serviced.  The  T-4  system  alleviated  this  problem,  but 
the  problem  illustrates  difficulties  that  can  arise  from  an  insufficient 
functional  analysis. 

The  T-4  system  in  fact  provides  another  illustration  of  this  problem. 

When  the  T-4  was  developed  the  F4  and  F106  represented  the  state  of  the 
art  in  interceptor  capabilities.  The  T-4  includes  flight  dynamics  models 
for  these  two  aircraft,  but  it  is  not  feasible  to  modify  the  T-4  to  model 
more  advanced  aircraft  (F15,  F16)  that  have  been  developed  since  then. 

As  a  result  students  are  often  surprised  by  the  extreme  responsiveness 
of  advanced  fighter  aircraft  when  they  see  them  for  the  first  time  in  live 
missions.  A  thorough  functional  analysis  would  have  noted  that  the  simulator 
should  be  capable  of  depicting  a  variety  of  aircraft,  and  the  data  base 
containing  aircraft  dynamics  models  should  be  conveniently  modifiable  by 
instructors  (who  are  not  computer  or  electronics  professionals)  as  developments 
occur. 

A  further  example  may  be  drawn  from  the  AWACS  simulator.  One 
responsibility  of  the  AWACS  surveillance  team  is  to  detect  and  respond  to 
electronic  warfare  (EW)  interference  with  the  radar  picture.  The  simulator 
is  capable  of  presenting  various  forms  of  degraded  imagery,  but  not  in  an 
interactive  manner.  The  ideal  situation  would  be  for  EW  interference  of 
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various  types  to  occur  when  requested  by  the  instructor.  The  time  required 
by  the  surveillance  team  to  detect  and  counteract  the  threat  would  then  serve 
as  measures  of  team  performance.  Such  an  approach  is  not  feasible  with 
the  present  system,  however,  because  EW  effects  cannot  be  switched  on 
at  will  and  operator  inputs  will  not  correct  the  problem  even  if  the  inputs 
are  appropriate. 

The  examples  above  focus  on  the  problem  of  identifying  relevant  problems 
for  individual  operators  (although  the  EW  example  applies  also  to  teams). 

A  final  example  illustrates  the  importance  of  including  team  functions  in 
the  functional  analysis.  The  AWACS  team  (mission  crew)  includes  surveillance, 
weapons,  support,  and  supervisory  personnel.  The  AWACS  simulator 
includes  consoles  for  all  crew  members  except  the  support  personnel 
(communications  operator,  radio  operator,  computer  display  maintenance 
operator,  airborne  radar  technician).  As  a  consequence,  important  coordina¬ 
tion  exercises  that  involve  these  crew  members  can  only  be  performed  in 
airborne  exercises- -the  cost  and  training  effectiveness  benefits  of 
simulation  cannot  be  applied  to  such  exercises. 

The  design  of  the  instructor  station  and  the  capabilities  provided  for  the 
instructor  to  interact  with  the  system  and  trainee  are  inadequate.  Little 
attention  was  given  to  any  kind  of  analysis  of  the  instructional  techniques  to 
be  used,  the  functions  of  the  instructor  in  using  them,  and  the  information 
he  would  need.  The  existing  instructor  station  is  over  the  student's  shoulder. 
The  lack  of  an  adequate  instructor  station  seriously  limits  the  effective  use 
of  the  training  device. 
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Partitioning  of  Functions 

After  the  individual  and  team  functions  of  a  system  have  been  identified 
and  defined  in  the  functional  analysis,  they  should  be  partitioned  into  two 
classes: 

•  Functions  to  be  trained  in  the  simulator 

•  Functions  to  be  trained  through  the  use  of  other  training 
media 


The  criteria  for  sorting  the  functions  in  this  way  are  complex  and  treated 
in  formal  ISD  documentation.  This  step  in  the  training  device  definition 
and  development  cycle  is  related  to  ISD  media  selection  analysis  procedures, 
which  are  intended  to  identify  the  most  cost-effective  training  medium  for 
each  operator  or  team  function.  Because  training  simulators  are  very 
expensive,  other  media  should  be  selected  whenever  possible.  This  will 
minimize  the  set  of  functions  to  be  simulated,  thus  reducing  development 
costs,  time,  and  technical  risk. 

The  main  point  of  the  partitioning  analysis  is  that  the  sorting  should  be 
based  on  training  criteria.  The  evidence  from  our  observations  and 
interviews  suggests  that  the  partitioning  has  historically  been  based  on 
other  criteria.  The  usual  practice  has  been  to  define  a  method  for  stimulating 
operational  equipment.  Functions  that  could  be  exercised  with  this  approach 
were  assigned  by  default  to  the  simulator  for  training.  The  problem  with 
this  approach  is  that  functions  that  could  potentially  be  trained  more 
effectively  with  other  media  are  trained  on  the  simulator  instead.  This 
has  the  unfortunate  effect  of  using  an  expensive,  scarce  resource  to  train 


functions  that  may  be  trained  more  effectively  in  other  ways.  In  addition, 
the  added  capability  required  to  simulate  such  functions  can  add  unnecessarily 
to  the  acquisition  and  life  cycle  costs  of  this  simulator. 

» 

Another  difficulty  with  the  historical  methods  for  partitioning  functions  has 
roots  in  the  failure  to  perform  adequate  functional  analyses.  This  is  the  ( 

increased  probability  that  functions  that  should  be  simulated  will  not  be 
because  they  were  not  identified  in 'the  functional  analyses. 

Definition  of  Training  Device  Requirements 

The  functional  requirements  for  a  training  simulator  may  be  grouped  into  » 

three  broad  categories: 

•  Operator  interface  functions 

•  Tactical  environment  models 

•  Training  support  functions 

Operator  interface  functions  include  operator- console  transactions.  These 
functions  are  identified  in  the  functional  analysis  and  have  typically  been 
well-defined  in  the  simulation  systems  we  observed.  Tactical  environment 
models  include  the  algorithms  and  data  base  depicting  capabilities,  numbers, 
and  geographical  distribution  of  Red  and  Blue  forces,  aerial  combat  tactics 
(engagement  models),  airspace  regulations,  ECM/ECCM  conditions, 
terrain  features,  weather  conditions,  time  of  day /night,  and  so  forth. 

These  models  should  be  defined  on  the  basis  of  the  functional  and  partitioning 
analysis.  The  models  were  defined  to  some  extent  in  the  models  we  observed, 
but  not  to  the  extent  deemed  necessary  by  the  instructional  developers, 
instructors,  and  students  we  interviewed. 
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The  first  two  categories  are  sufficient  for  defining  a  simulator  but  not 
a  training  device.  A  training  simulator  is  a  part  of  a  larger  training 
program,  and  it  should  therefore  include  features  that  are  designed 
explicitly  to  support  training  functions.  Examples  of  such  functions  are: 

•  Automated  assessment  and  monitoring  of  operator  and 
team  performance 

•  Presentation  of  performance  data  to  instructors 

•  Automated  branching  among  lesson  segments  on  the  basis 
of  student  of  team  performance 

•  Automated  delivery  of  feedback  and  prompting  to  students 
and  teams 

•  Capability  for  simulating  events  in  real  time  and  at  rates 
other  than  real  time 

•  Capability  for  replaying  simulated  events 

•  Part-task  training  capabilities  —  the  ability  to  exercise  a  subset 
of  the  operator's  or  team's  duties 

•  Capability  for  presenting  successive  approximations  to  the 
quality  and  appearance  of  imagery  on  a  display  scope 

•  Flexibility,  ease  of  maintenance,  and  convenient  modifiability 
so  that  instructors  who  are  not  computer  professionals  can 
make  necessary  changes  in  the  data  base  and  models  driving 
exercise  scenarios 
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The  first  four  features  are  nontrivial  in  that  they  require  advances  in  the 
state  of  the  art  in  performance  measurement.  This  problem  is  especially 
serious  in  the  context  of  team  training.  The  next  several  features  pose 
difficult  software  problems,  but  they  are  problems  that  have  been  solved  ' 

in  other  simulation  and  CAI  applications.  The  final  feature  references  a 
general  need  for  human  factors  analyses  directed  at  improving  the  interface 
between  the  instructor  and  the  training  system.  The  set  of  features  is  intended 
to  emphasize  that  the  simulator  or  training  device  should  be  a  tool  for 
instructors  to  use  in  achieving  the  training  objective  of  a  training  program. 

It  is  not  sufficient  for  a  simulator  to  be  just  a  faithful  replica  of  the 
operational  system — it  must  also  support  training  functions  per  se. 

0 

None  of  the  simulators  we  observed  included  significant  training  support 

functions.  Some  of  the  systems  included  consoles  for  intercept  pilot 

simulators  (for  example,  T-4  drivers),  but  these  people  are  not  involved  in  » 

the  planning,  control,  or  assessment  of  training.  In  the  manual  training 

system  at  Tyndall,  for  example,  the  instructor  must  literally  stand  behind 

the  student  who  is  working  the  scope,  and  he  must  watch  the  student's 

every  move  (especially  during  early  training).  This  level  of  instructor 

attention  may  be  appropriate  during  live  missions  when  safety  is  of  / 

paramount  concern,  but  many  of  the  instructor's  functions  could  be  automated 

in  simulation  exercises.  This  would  free  the  instructor  to  perform  more 

important  duties.  The  equipment  at  Tyndall  is  old,  but  newer  simulators 

follow  the  same  approach.  As  one  course  developer  remarked  about  the 

AWACS  simulator,  the  newest  of  the  systems  we  observed,  the  instructors 

must  still  rely  on  just  "their  eyeballs  and  a  piece  of  paper"  in  assessing 

and  recording  individual  and  team  performance. 
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Development  of  a  Training  Device 


Training  device  development  follows  directly  from  the  preceding  analysis 

and  definition  steps.  This  is  largely  a  software  and  hardware  engineering 

problem  and  does  not  require  discussion  here  except  to  recommend  that 

training  and  human  factors  specialists  should  be  kept  in  the  loop  throughout 

the  design  process.  The  purpose  of  the  participation  is  to  monitor  the 

design  process,  provide  inputs  for  decisions  that  affect  the  student /system, 

instructor/system,  and  student /ins  true  tor  interfaces,  and  defend  the 

training  and  human  factors  philosophy  when  necessary.  The  problems 
2 

with  current  C  simulators  demonstrate  the  need  for  input  from  these 
specialists  early  and  on  a  continuing  basis  during  the  simulation  development 
process. 
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CHAPTER  HI 


STATUS  OF  AFC2T2  PROGRAMS: 

IMPLEMENTATION  AND  MANAGEMENT 

* 


OVERVIEW 

The  success  of  the  training  mission  is  determined  by  the  relevance  of  the 

skills  trained  to  the  skills  required,  the  effectiveness  of  the  methods  used 

to  conduct  and  evaluate  training,  and  program  management.  The  survey  of 
2 

TAC  training  revealed  both  strengths  and  weaknesses  in  the  skills  trained, 
methods  of  training  and  assessment,  and  program  management.  Some  of 
the  weaknesses  follow  from  weaknesses  of  training  definition  and  develop¬ 
ment.  Others  are  unique  to  the  way  in  which  training  is  conducted  and 
managed. 

3  2 

Discussion  of  the  strengths  and  weaknesses  in  AFC  T  implementation  and 
management  is  organized  around  five  topics.  The  skills  trained  and  program 
management  are  treated  separately,  and  methods  of  training  and  assessment 
are  divided  into  three  parts.  This  chapter  is  therefore  divided  into  five 
sections,  as  follows: 

•  Individual  and  team  skilis  trained 

•  Instructional  strategies  and  sequencing 

•  Simulation/ simulator  features  and  utilization 
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Assessment  of  team  and  system  performance 


•  Program  management  issues 
A  brief  summary  of  the  contents  of  each  section  follows. 

2 

The  individual  and  team  skills  being  trained  in  C  training  programs  are 
described  in  the  first  section.  It  is  argued  that  team  skills  as  properties 
of  the  team  as  a  whole  do  not  exist.  Team  skills  are  best  understood  as 
individual  cognitive  skills  required  for  carrying  out  interactions  with  other 
team  members.  The  training  of  team-oriented  cognitive  skills  is 
unstructured  and  unsystematic,  because  the  requirement  to  train  them  is 
not  made  clear  during  program  definition  and  development. 

2 

The  important  instructional  methods  of  C  training  are  the  master-apprentice 

model  and  role-playing.  The  master-apprentice  model  has  strengths  and 

weaknesses  that  were  identified.  The  latter  are  especially  evident  in 

informal  training,  that  is,  OJT.  Role-playing  techniques  are  used  to  good 

advantage  in  formal  training  programs.  There  is  little  sequencing  of 

training  individual  and  subteam  skills,  as  subteam  skills  are  fundamental 
2 

to  the  C  console  operator  tasks,  especially  those  of  the  Air  Weapons 
Controller  (AWC).  Team  and  superteam  skill  training  follow  individual  and 
subteam  skill  training,  but  are  themselves  not  carried  out  sequentially. 

The  AWACS  training  program  is  subject  to  the  most  sequencing  problems 
as  a  result  of  mission  variability  and  uncertainty.  That  is,  given  the 
worldwide  potential  commitment  of  AWACS  and  the  multitude  of  possible 
mission  scenarios,  it  is  difficult  to  determine  which  system  exercises  or 
simulated  combat  missions  to  train  initially,  and  so  on. 


2 

Simulators  and  simulation  are  used  extensively  in  AFC  training  programs. 

2 

The  invaluable  role  of  simulation  in  AFC  training  was  acknowledged  by  all 
survey  respondents.  However,  several  serious  problems  prevent  simulation 
from  being  as  effective  as  it  could  be.  Current  simulations  lack  tactical  ► 

realism  in  terms  of  the  characteristics  of  the  projected  threat  and  in 
war-gaming,  that  is,  interactive  capability.  Geopolitical  relevance  of 
simulations  based  on  past  conflicts  is  low.  Simulators  lack  features  and  1 

capabilities  supportive  of  the  instructor  interface. 

The  assessment  of  team  readiness  and  system  effectiveness  is  difficult, 
incomplete,  and  hampered  by  non-training  factors,  like  the  unavailability 
of  live  flying  resources  or  the  inability  of  current  simulations  to  model  a 
European  theatre  conflict.  The  conditions  under  which  teams  are  evaluated 
vary  widely. 

» 

Individual  team-oriented  skills  are  assessed  in  a  global  fashion  only,  but 
the  team  performance  process  is  observed  and  feedback  based  on  these 
observations  is  delivered  both  orally  and  in  writing.  System  effectiveness 
measures  do  not  take  the  difficulty  of  exercises  into  account.  Political 
pressure  sometimes  keeps  operational  problems  from  surfacing.  J 

2 

Finally,  issues  of  program  management  concern  problems  in  C  career 

fields,  resulting  in  low  retention  rates  and  the  steady  loss  of  experienced  * 

individuals.  There  also  are  problems  specific  to  specific  programs: 

Training  managers  do  not  appear  to  have  an  adequate  understanding  of  the 
2 

C  training  pipeline,  and  the  training  implemented  for  newly  acquired 
systems  does  not  always  address  the  needs  of  inexperienced  personnel. 
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INDIVIDUAL  AND  TEAM  SKILLS  TRAINED 


\ 


I 


It  is  generally  agreed  in  contemporary  psychology  that  there  are  two  broad 

categories  of  skill,  namely,  perceptual- motor  and  cognitive  (References  17 

and  18).  Both  of  these  categories  can  be  more  finely  divided.  The  survey 
2  2 

of  AFC  T  indicated  that  the  following  types  of  skills  are  being  trained 
either  formally  or  informally: 


Perceptual-  Motor 
Sensory 
Perceptual 
Manipulative 


\ 

Procedural 


Cognitive 
Computational 
Communication 
De  cision-  Ma  king 
Problem-Solving 


Perceptual-motor  skills1  are  required  in  the  detection  and  interpretation  of 
'  radar  imagery  and  in  operation  of  the  display  console.  Procedural  skills 

are  a  hybrid  of  perceptual- motor  and  cognitive  skills;  they  are  required  to 
properly  operate  equipment  and  to  perform  any  task  involving  a  preferred 
sequence  of  events  and  actions.  Cognitive  skills  involve  the  mental  process 
ing  of  information  and  knowledge  and  are  required  in  tasks  that  demand 
•  thinking. 


Cognitive  skills  form  the  foundation  for  efficient  and  effective  team  perform 
2 

ance.  In  C  tasks  the  paramount  requirement  is  the  timely  delivery  of 
information  to  a  teammate.  The  particular  information  that  gets 


1  2 

Knowledge  of  the  C  system.  Air  Force  and  TAC  policies,  procedures  and 
tactics,  and  the  threat  is  a  prerequisite  to  acquisition  of  skills. 
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communicated  usually  is  a  product  of  other  cognitive  skills  like  computation, 
decision-making,  and  problem-solving.  It  is  the  timing  of  one's  own 
actions  and  the  anticipation  of  a  teammate's  needs  or  actions  which  produces 
the  coordinated  behavior  of  a  team.  , 

An  individual's  cognitive  skills  are  the  only  ones  identifiable  as  team  skills. 

Team  performance  is  a  byproduct  of  the  execution  of  individual  skills  in  a  1 

team  context.  As  long  as  every  individual  team  member  is  proficiently 
performing  team-oriented  skills,  the  team  performance  that  is  observed 
wiil  be  proficient.  Because  of  the  interdependencies  in  team  performance, 
if  one  individual  is  unskilled,  his  teammates  wiil  have  to  adapt,  or  team 
performance  will  suffer. 

# 

One  of  the  implications  of  this  view  for  identification  of  a  "good"  team  is 
that  assessment  should  proceed  on  two  levels.  First,  tests  of  individual  ( 

team-oriented  behavior  should  be  developed,  and  second,  it  is  important 
to  focus  on  the  process  in  evaluating  team  readiness. 

Also  according  to  this  view,  team-oriented  skills  at  times  compete  with 

perceptual- motor  or  procedural  skills  for  the  attention  of  the  individual. 

Thus,  another  aspect  of  skilled  performance,  time  sharing,  is  important 
2 

in  C  tasks.  The  time-sharing  requirement  of  team-oriented  skill  may 

account  for  the  observed  decrement  in  individual  proficiency  that  occurs  ' 

when  an  individual  performs  in  a  team  context. 

It  wiil  be  helpful  to  provide  some  examples  of  how  these  types  of  skills 

2 

characterize  the  tasks  and  duties  of  the  three  C  personnel  categories. 
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Console  Operators  and  Support  Personnel 

The  primary  source  of  job- relevant  information  for  surveillance,  height, 
tracking,  air  traffic  control,  airborne  radar  and  weapons  technicians, 
and  surveiilance/RICM  officers  and  weapons  controllers  is  a  display  of 
radar  imagery.  The  imagery  may  be  either  raw  or  processed,  depending 
on  the  type  of  system- -manual  or  automatic.  To  the  untrained  eye  the 
data  presented  in  such  a  display  are  at  best  so  many  stationary  or  moving 
blips.  To  the  trained  eye  this  data  is  information  indicating  the  relative 
positions  and  movements  of  airborne  objects.  The  display  presents  a 
"God's-eye  '  view  of  activity  in  a  particular  airspace.  ATC  instructors  in 
the  Air  Weapons  Controller  Fundamentals  course  state  that  it  is  difficult 
for  students  to  attain  this  "looking  down"  perspective  on  the  display. 

The  surveillance  and  tracking  technicians  must  learn  how  to  detect  data 

representing  actual  aircraft  so  that  the  aircraft  may  be  identified,  clas- 

2 

sified,  and  their  movements  tracked.  In  automatic  C  systems,  the 
technician  must  manipulate  a  control,  for  example,  a  trackball,  to 
inform  the  computer  of  the  location  at  which  a  target  has  been  detected. 
The  difficulty  of  the  detection  task  is  compounded  by  the  long  hours  which 
may  be  spent  on-station;  vigilance  is  hard  to  maintain.  This  difficulty  is 
more  characteristics  of  SAGE  (JSS)  and  a  deployed  control  and  reporting 
center/post  (CRC/CRP)  than  of  AWACS,  which  stays  on-station  as  long 
as  fuel  is  available.  Height  and  tracking  technicians  learn  how  to  extract 
information  from  the  display  regarding  height,  speed,  and  direction  of 
movement.  These  tasks  are  supported  by  the  system  computer. 
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The  sensory  and  perceptual  skills  of  the  surveillance/ RICM  officer,  and 
weapons  controller,  also  are  applied  to  display  data,  but  the  tasks  require  a 
higher  degree  of  pattern  recognition  than  those  of  other  console  operators. 

In  the  case  of  surveillance /RICM  officers,  types  and  patterns  of  electronic  > 

jamming  or  other  interference  must  be  recognized  (and  defeated).  Weapons 
controllers  must  recognize  tactics  of  target  tracks  and  discern  trends  in 
movement  of  friendly  and  hostile  tracks.  Although  these  sensory  and 
perceptual  skills  are  not  easily  acquired,  there  were  no  indications  of 
special  problems  beyond  the  initial  difficulty  some  students  have  adopting 
a  God's-eye  view  of  the  display. 

Manipulative  skills  for  both  job  positions  involve  the  use  of  input  devices 
like  trackballs,  keyboards,  or  push  buttons  to  keep  the  system  computer 
current  and  to  call  up  various  display  aids. 

• 

The  sensory  and  perceptual  tasks  of  console  operators  are  complicated  by 

two  additional  factors:  amount  of  air  traffic  and  familiarity  of  airspace 

location.  The  speed  and  accuracy  of  manipulative  tasks  are  complicated 

by  hardware  (for  example,  AWACS  keyboard)  and  software  (for  example, 

number  of  inputs  required)  design.  , 

In  SAGE/JSS  the  volume  of  air  traffic  is  somewhat  dependent  on  the  region 
but  tends  to  be  uniformly  high.  A  CRC/CRP  or  AWACS  is  subject  to 
varying  amounts  of  air  traffic  depending  upon,  among  other  things,  the 
tactical  situation  and  deployment  location.  In  sum,  SAGE/JSS  has  high  air 
traffic  volume,  but  a  stable  location  with  a  familiar  background  of  displayed 
noise;  on  the  other  hand,  TACS  elements  and  AWACS  have  unpredictable 
air  traffic  volume,  and  changing  locations  requiring  adeptness  at  filtering 
out  irrelevant  radar  data. 
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As  for  manipulative  tasks,  good  system  design  can  make  training  and 
operation  much  simpler.  The  selection  of  functions  to  be  performed  by 
man  versus  machine,  and  the  representation  of  machine  functions  in  terms 
of  keyboard  layout  and  number  of  switch  actions,  both' impact  the  ease  of 
skill  acquisition  and  the  quality  of  performance.  For  example,  some  table 
updates  in  AWACS  could  be  accomplished  by  machine;  without  a  technician 
the  controller  is  occasionally  too  busy  to  enter  tabular  information.  AWACS 
also  provides  an  example  of  the  training  impact  of  a  poor  keyboard  layout 
(see  Chapter  3  Overview).  The  profusion  of  TSQ-91,  OJ-108  console 
function  switches,  and  their  occasionally  incompatible  arrangement,  places 
a  critical  limit  on  the  speed  of  performance. 

Procedural  skills  are  required  in  all  console  operator  positions.  Training 

for  procedural  tasks  is  easily  developed  and  such  skills  are  readily  acquired. 

Procedural  performance  is  rapid  and  accurate  provided  that  the  task  occurs 

with  regularity.  Infrequent  procedural  tasks  are  typically  performed  less 

rapidly  and  accurately  than  frequent  ones.  One  example  of  this  skills 

2 

maintenance  problem  was  found  in  AWACS  training.  Interservice  C  system 
exercises  have  not  been  frequent.  Consequently,  such  exercises  are 
characterized  by  poor  performance  of  procedures  for  linking  AWACS  to 
Navy  elements  and  aircraft  using  tactical  data  link  A  (TADL-A). 

Procedural  tasks  requiring  timely  reporting  of  information  to  teammates 
or  supervisors  (for  example,  during  an  aircraft  handover)  involve  the 
acquisition  and  performance  of  team-oriented  skills.  Little  is  known  about 
the  decay  of  these  skills,  but  the  necessary  anticipation  or  timing  is 
observed  to  be  lost  first  while  the  sequence  of  actions  is  remembered  for 
long  periods  of  time.  The  decay  of  team-oriented  skills  is  very  probably 
one  of  the  sources  of  ineffective  team  performance. 
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The  tasks  and  duties  of  C  system  technicians  (except  weapons)  do  not 
depend  heavily  on  cognitive  skills,  although  there  are  requirements  in  the 
areas  of  decision-making  and  communication.  Important  or  unusual 
information  should  be  brought  to  the  attention  of  the  technician's  supervisor 
or  other  team  member.  The  criteria  for  taking  this  action  are  not  always 
clearcut  and  may  change  as  a  function  of  emergent,  unpredictable  properties 
of  the  task  environment.  There  is  no  significant  requirement  to  develop 
and  maintain  mathematical  or  significant  problem-solving  skills. 

Cognitive  skill  acquisition  and  maintenance  is  critical  to  successful 
performance  by  weapons  controllers /technicians  and  surveillance /RICM 
officers.  Furthermore,  these  skills  are  almost  always  team-oriented  and 
have  a  tremendous  impact  on  team  performance. 

Computational  skills  are  required  to  solve  problems  of  algebra  and 
geometry.  Such  computations  are  necessary  to  provide  information  that 
will  yield,  for  example,  an  intercept  course  heading  or  speed.  It  is  useful 
to  distinguish  between  mathematical  and  computational  skills,  where  the 
former  are  basic  to  a  wide  range  of  situations  and  the  latter  are  those 
required  to  solve  task- specific  problems.  Mathematical  skills  then  are 
the  foundation  for  computational  skills.  Respondents  in  the  survey 
commented  that  it  was  difficult  to  train  computational  skills  because  many 
of  the  students  did  not  possess  the  requisite  mathematical  skills. 


Communications  skills  are  essential  to  any  interpersonal  interaction.  They 
vary  in  complexity  from  learning  and  using  a  brevity  code  in  radio  trans¬ 
missions  to  speaking  and  listening  or  writing.  These  skills  are  vital  to 
any  team  performance,  especially  the  team  formed  by  the  weapons  controller, 
technician,  and  pilot.  The  controller  must  communicate  certain  information 
to  the  pilot  at  the  right  time  and  in  the  right  way.  The  technician  must 
provide  information  to  the  controller,  or  other  technicians,  meeting  these 
same  criteria. 

The  particular  information  that  is  passed  to  the  pilot  by  the  controller 
depends  upon  many  factors.  Determining  what  information  to  communicate 
to  the  pilot  demands  decision-making  and,  occasionally,  problem-solving 
skills  on  the  part  of  the  controller;  good  controlling  is  more  a  matter  of 
technique  than  procedural  skill.  For  example,  effective  control  of  aircraft 
requires  anticipation  of  information  needs  based  upon  the  current  position, 
the  desired  position,  the  rate  of  change,  the  type  of  intercept,  and  so  on. 
Improving  the  air  picture  is  a  problem  that  all  surveillance /RICM  officers 
must  solve.  Problem-solving  and  decision-making  come  into  play  when 
non-standard  or  unfamiliar  situations  occur  and  available  standard 
procedures  do  not  apply.  The  individual  must  then  diagnose  the  situation, 
identify  a  goal  or  objective  which  will  be  responsive  to  the  needs  of  the 
situation,  adapt  his  skills  to  devise  a  sequence  of  actions  to  accomplish  it, 
and  then  execute  that  sequence.  He  will  often  have  to  do  this  thinking  and 
planning  concurrently  with  executing  the  action.  These  behaviors  are 

characteristic  of  the  demands  placed  on  individuals  in  emergent  situations. 

2 

The  emergent  nature  of  the  tactical  and  air  defense  C  environment 
creates  the  need  to  train  such  higher-order  cognitive  skills. 
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No  team  can  be  effective  unless  the  individual  team  members  are  highly 

practiced  and  skilled.  Furthermore,  team-oriented  individual  skills  like 

communication,  decision-making,  and  problem-solving  are  critical  to  the 

2 

acquisition  of  team  skills,  which  in  C  systems  are  primarily  related  to  the 
interdependence  between  and  among  individuals  for  the  timely  transmission 
and  receipt  of  correct  information.  Weaknesses  in  training  these  individual 
team-oriented  skills  negatively  impact  team  skill  acquisition. 

First- Line  Supervisors  and  Battle  Staff 

The  individuals  trained  for  these  positions  already  possess  (or  are  assumed 
to  possess)  the  skills  of  console  operators  and  support  personnel  in  their 
career  field.  The  important  individual  skills  are  cognitive,  with  manage¬ 
ment  and  leadership  skills  being  added  to  those  previously  identified. 

Skilled  team  supervisors  and  staff  are  critical  to  effective  team  performance. 
For  example,  a  Senior  Director  on  AWACS  or  a  Weapon  Assignment 
Officer  in  a  CRC/CRP  must  ensure  that  the  work  distribution  among 
controllers  (directors  in  AWACS)  is  equitable.  Higher- level  supervisors 
and  staff  are  likewise  responsible  for  the  management  of  various  resources 
to  ensure  that  no  portion  of  the  team  is  overloaded  or  overcommitted. 

Thus,  in  the  team  environment  ail  individual  skills  of  supervisors  and  staff 
should  be  viewed  as  team-oriented  skills  without  which  the  team  will  not 
perform  effectively.  Any  weaknesses  in  training  these  skills  are  reflected 
in  team  performance. 
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Current  practice  in  training  supervisors  and  battle  stuff  is  informal, 
unplanned  in  terms  of  training  objectives,  and  unsystematic  in  coverage. 

No  formal  training  is  defined  and  developed  for  these  positions.  The 
learning  by  an  individual  is  largely  a  function  of  the  richness  of  the  exercises 
so  that  he  can  be  exposed  to  a  wide  variety  of  representative  events,  and  his 
perceptiveness  in  attending  to,  recognizing,  and  remembering  relevant 
things.  This  situation  is  very  chancy.  The  acquisition  of  team-oriented 
skills  will  be  haphazard  and  incomplete  as  long  as  this  informal, 
unsystematic  approach  is  used. 

INSTRUCTIONAL  STRATEGIES  AND  SEQUENCING 

2 

The  formal  methods  used  in  TAC  training  vary  in  terms  of  the  technique 

of  instruction  and  the  conditions  of  practice.  The  major  instructional 

technique  follows  the  master- apprentice  model;  another  important 

2 

technique  used  is  role-playing.  The  best  T  technique  in  use  consists  of 
intrateam  briefings.  The  conditions  of  practice  include  simulated 
intercepts,  simulated  combat  missions,  and  system  exercises  varying  in 
scope  and  complexity.  OJT  is  an  example  of  an  informal  training  method 
that  depends  critically  upon  the  conditions  of  practice.  There  is  some 
sequencing  of  individual  and  team-oriented  training  in  initial  training 
programs,  but  there  is  little  sequencing  of  training  in  operations  training 
programs. 

Master- Apprentice  Model 

The  master-apprentice  model  is  the  dominant  instructional  technique  in 
2 

TAC  training.  It  is  used  both  in  formal  and  informal  training  for  all 
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personnel  categories.  The  model  is  characterized  by  an  instructional 
method  consisting  of  three  steps:  1)  task  explanation  and  demonstration  by 
the  master,  2)  performance  by  the  apprentice,  and  3)  feedback  to  the 
apprentice  based  on  the  master's  observations. 

This  method  is  necessary  for  training  skills,  in  general,  and  cognitive 
skills  and  technique,  in  particular.  However,  the  effectiveness  of  the 
method  is  limited  by  certain  inherent  problems  which  may  be  exacerbated 
by  factors  related  to  the  training  environment  and  structure. 

The  model  assumes  that  the  master  is  an  expert  performer,  and,  further¬ 
more,  that  he  has  knowledge  about  the  important  aspects  of  his  tasks  and 
performance.  Expert  performers  are  recognized  by  other  experts  fairly 
readily,  although  the  subjective  nature  of  this  evaluation  has  its  own 
shortcomings.  Unfortunately,  experts  do  not  always  have  insight  into 
how  they  do  what  they  do.  Nor  do  all  who  occupy  expert  positions  have 
expert  qualifications.  Some  aspects  of  expert  performance  may  not  be 
accessible  to  examination. 

The  accessible  aspects  of  an  expert's  performance  must  be  communicated 
to  the  apprentice  in  such  a  way  that  the  apprentice  can  learn.  In  short,  the 
expert  must  be  able  to  instruct,  and  do  this  regardless  of  the  conditions 
of  practice  and  the  structure,  stage,  or  setting  of  training.  These  require¬ 
ments  pose  few  problems  in  ATC  resident  training,  where  instructors 
receive  special  training;  the  structure  is  fairly  well  developed,  (that  is, 
lesson  plans)  and  the  conditions  of  practice  are  relatively  straightforward. 
However,  in  TAC  resident  and  operations  training  programs,  where 
instructors  may  have  no  training  in  instructional  techniques  and  the 
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structure  and  conditions  are  highly  variable,  the  master- apprentice 
model  can  be  ineffective  or  negative.  In  any  setting,  controller  training, 
because  of  the  importance  of  technique,  is  difficult  to  standardize;  each 
instructor  stresses  different  things  dependent  on  his  own  technique. 

OJT  in  a  live  environment  represents  a  worst- case  example  of  the  problems 
that  can  be  caused  by  applying  the  master-apprentice  method  in  an 
unstructured  setting.  AWACS  Mission  Crew  Commander  (MCC)  student 
respondents  commented  that  the  duties  of  their  position  are  not  clearly 
established  and  consequently,  MCC  instructors  teach  different  job  tasks 
dependent  upon  their  own  experience.  Because  of  the  importance  of 
leadership  and  management  skills,  this  lack  of  standardization,  which  can 
be  expected  to  exist  in  any  OFT  setting  where  training  material  is  lacking, 
can  cause  severe  problems  in  team  (especially  soft  teams)  performance. 

As  one  SAGE  student  Senior  Director  put  it,  "Trying  to  learn  how  to  be 
a  Senior  Director  from  a  manual  is  like  trying  to  learn  English  by  reading 
the  dictionary. " 

Role-Playing 

This  method  of  instruction,  in  which  students  alternate  between  different 
positions,  is  used  to  good  advantage  in  initial  training  programs.  In  the 
Air  Weapons  Controller  Fundamentals  course,  students  work  together  in 
groups  of  two  or  three.  They  take  turns  being  the  Weapons  Technician 
and,  as  a  consequence,  learn  the  duties  of  that  team  member.  This 
technique  is  also  used  in  the  manual  SQT  and  APQ  courses.  In  the  funda¬ 
mental  Aerospace  Warning  and  Control  Technician  courses  students  work 
in  groups  of  four  and  rotate  among  the  positions  of  surveillance  scope 
operator,  teller,  plotter,  and  recorder. 

Ill 


The  only  possible  drawback  in  using  this  technique  might  be  for  training 

AW  ACS  Weapons  Directors  who  work  without  Weapons  Technicians. 

However,  student  WD  respondents  indicated  no  problems  arising  from  this 

2 

practice.  This  instructional  technique  is  a  strength  in  TAC  initial 
training  programs. 

Intrateam  Briefings 

2 

A  very  effective  technique  for  accomplishing  T  is  the  use  of  intrateam 
briefings.  This  technique  is  used  in  the  SAGE  and  AWACS  training  programs. 
Intrateam  briefings  consist  of  presentations  by  one  section,  for  example, 
surveillance,  to  the  rest  of  the  team  regarding  that  section's  role  and 
responsibilities.  These  presentations  are  independent  of  pre-  or  post¬ 
mission  briefings.  Such  presentations  can  do  much  to  enhance  team 
awareness  and  structure  a  shared  plan,’  both  critical  elements  of  team 
performance.  Intrateam  briefings  were  not  observed  during  the  survey. 

The  specific  nature  of  those  presentations  is  therefore  not  known.  It  is 

2  2 

our  opinion  that  frequent,  well- structured  briefings  are  an  AFC  T  program 
strength. 

Conditions  and  Sequencing  of  Training  and  Practice 

2 

The  basic  job  tasks  of  C  personnel  are  not  in  and  of  themselves  inordinately 

complex.  Rather,  it  is  the  nature  of  the  operational  environment  in  which 

the  tasks  must  be  performed  that  makes  them  difficult.  Theater  operations 

are  characterized  by  unpredictable  events  and  matters  of  life  or  death.  The 
2 

goal  of  C  training  is  to  prepare  individuals  and  teams  to  function  efficiently 
in  such  environments.  Even  the  conditions  of  a  live-flying  large-scale 
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exercise  environment  can  only  approximate  the  conditions  of  a  theater 
operation.  In  this  sense,  all  military  training  is  carried  out  under  conditions 
of  degraded  fidelity,  but  some  conditions  can  be  made  more  realistic  than 
others. 

A  strategy  for  determining  and  implementing  the  conditions  of  training  must 

not  only  specify  the  level  of  fidelity  of  various  environmental  dimensions, 

but  must  also  specify  the  order  in  which  different  performance  contexts  are 

to  be  experienced.  If  the  strategy  fails  to  define  conditions  and  sequencing, 

team-oriented  skill  acquisition  will  be  haphazard  and  the  quality  of  team 

performance  in  an  operational  environment  will  be  highly  variable. 

2 

Conditions  and  sequencing  of  C  training  and  practice  are  not  always  well 
implemented. 

2 

As  discussed  earlier,  the  C  training  performance  context  may  be  individual, 
subteam,  team,  or  superteam.  Team-oriented  skills  are  important  in  all 
but  the  individual  context.  Both  the  emphasis  on  and  level  of  team-oriented 
skills  change  from  one  performance  context  to  another  more  inclusive  one. 
Communication  is  a  paramount  subteam  skill,  but  decision-making  and 
problem  solving  tend  to  be  more  critical  in  team  and  superteam  contexts. 

The  dimensions  of  performance  context  which  affect  the  level  of  skill 
required  include  the  type  of  environment,  live  or  simulated;  the  number, 
characteristics,  and  missions  of  aircraft  (resources)  being  controlled; 
the  amount  of  ECM  and  communications  jamming;  and  the  number,  flexibility, 
and  tactics  of  hostile  aircraft. 


In  initial  training  the  complexity  along  these  dimensions  is  minimal,  but  is 
sequenced  and  increased  within  each  course.  For  example,  AWC  subteam 
skills  are  first  trained  in  a  simulated  intercept  environment  with  one 
friendly  and  one  hostile  (referred  to  as  a  1  v  1  intercept),  and  predetermined 
tactics.  When  the  student  completes  an  initial  progress  check,  he  moves 

on  to  a  live  1  v  1.  As  training  continues  there  is  no  substantive  increase 
in  complexity  of  simulated  and  live  intercepts.  At  the  completion  of 
fundamental  SQT  the  student  can  safely  perform  most  parts  of  a  live 
1  v  1  intercept  using  different  tactics.  The  primary  team-oriented  skill 
which  has  been  trained  is  communication;  there  is  a  minimum  of  decision¬ 
making  and  problem-solving  needed  at  this  stage.  AFQ  training  provides 
additional  practice  on  subteam  communication  skills,  and  increases 
intercept  difficulty  to  2  v  1.  At  the  completion  of  training  the  student  can 
safely  perform  a  live  2  v  1  intercept  with  some  prompting  on  how  and  when 
to  perform  the  task,  especially  in  non-standard  situations. 

The  details  of  team  and  superteam  training  conditions  and  sequencing  differ 

2  , 

according  to  the  TAC  system.  For  example,  the  CRC/CRP-bound  student 
controller  receives  additional  practice  on  subteam  skills  at  the  607  TCTS 
before  entering  an  operations  training  program  to  practice  in  team  and 
superteam  contexts.  On  the  other  hand,  the  AWACS  student  controller 
begins  practice  in  a  team,  and  occasionally  a  superteam,  context  shortly 
after  beginning  initial  transition  training  at  the  966  AWACS  Training 
Squadron;  subteam  training  continues  during  this  period  as  well. 
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The  operations  training  program  which  follows  for  both  systems  is  primarily 
concerned  with  training  team  and  superteam  skills.  However,  subteam 
skill  training  also  takes  place,  because  each  higher- level  context  includes 
all  those  below  it;  all  individual  and  team-oriented  skills  are  practiced  in 
a  superteam  context. 

The  conditions  of  controller  subteam  skill  training  are  presented  as  part 
of  a  live  or  simulated  intercept  event.  The  conditions  of  controller  team 
and  superteam  training  are  presented  as  part  of  a  simulated  combat  mission, 
that  is,  exercise,  which  may  or  may  not  include  live  flying. 

Exercises  are  more  difficult  to  define  and  implement  than  intercept  events. 
This  difficulty  is  compounded  by  mission  variability  and  shortage  of  live 
intercept  events.  The  SAGE  wartime  mission  is  relatively  straightforward: 
exercises  for  SAGE  thus  are  spinoffs  of  a  single  prototype.  The  TACS 
wartime  mission  is  complex  and  the  development  and  implementation  of 
exercises  must  take  this  complexity  into  account.  AWACS  represents  a 
worst-case  example  of  mission  variability  which  produces  a  correspondingly 
large  number  of  possibilities  for  simulated  combat  missions. 

The  number  and  complexity  of  exercises  is  subject  to  resource  limitations. 
Consequently,  it  is  impossible  for  AWACS  team  and  superteam  training  to 
be  adequately  representative  of  possible  mission  requirements.  Further¬ 
more,  the  sequencing  of  this  training  is  unsystematic.  AWACS  "piggy¬ 
backs"  on  scheduled  SAGE  and  TACS  exercises  when  possible  or  arranges 
superteam  training  when  necessary  to  achieve  operational  objectives, 
for  example,  the  development  of  an  Airborne  Tactical  Control  System 
(ATACS,  AWACS,  and  ABCCC),  or  linking  with  the  Navy  via  TADL- A. 


Thus,  individual  and  team  participation  in  any  particular  exercise  is  a 
function  of  timing  rather  than  skill  level  achieved.  Because  the  AWACS 
simulator,  as  discussed  in  the  next  section,  does  not  have  a  full  team 
simulation  capability,  it  is  not  capable  of  providing  a  solution  to  this 
problem.  This  problem  is  compounded  by  the  lack  of  hard  crews  in 
AW  ACS.  Individuals  who  have  seldom  worked  together  before,  who 
therefore  lack  certain  team-oriented  skills,  may  find  themselves  in  a 
situation  requiring  superteam  skills.  This  provides  for  a  suboptimal 
training  experience. 

These  types  of  problems  are  evident  in  TACS  to  a  lesser  degree  than  AWACS, 
but  they  still  exist.  Limited  numbers  and  complexity  of  exercises  pose  a 
representativeness  problem,  but  full  team  simulation  capability  mitigates 
this  somewhat.  Turnover  in  personnel  may  make  individuals  subject  to 
sequencing  which  is  inconsistent  with  their  level  of  experience. 

SAGE  is  probably  in  the  best  shape  on  this  issue  because  of  its  well- 
defined  wartime  role  and  long  history  of  fine-tuning  exercise  conditions. 
Sequencing  for  individual  team  members  can  still  pose  some  problems, 
however. 

SIMULATION/ SIMULATOR  FEATURES  AND  UTILIZATION 

2 

Simulators  and  simulation  exercises  are  used  extensively  in  AFC 

training  programs.  The  training  devices  are  used  for  initial  and  advanced 

2 

training  of  individual  operators  and  C  teams.  All  participants  we  inter¬ 
viewed- -instructional  program  managers /developers,  instructors,  and 
students--agreed  that  simulation  plays  an  invaluable  role  in  current 


programs.  The  participants  also  agreed,  however,  that  serious  problems 

prevent  simulation  from  being  as  effective  as  it  could  be.  These  problems 

have  their  roots  in  the  training  device  definition  and  development  process 

(see  Chapter  2),  and  have  a  direct  impact  on  the  quality  (cost  effectiveness) 

2 

of  AFC  programs.  The  following  points  summarize  the  concerns  of  the 
people  we  interviewed: 

•  Insufficient  tactical  realism 

•  Low  geopolitical  relevance 

•  Lack  of  training  support  functions 

•  Inability  to  effect  improvements  in  the  training  devices 

The  problem  of  insufficient  tactical  realism  takes  two  forms.  The  first  is 
that  no  current  simulation  exercises  match  the  projected  threat  either  in 
terms  of  numbers  or  capabilities.  Live  exercises  are  even  less  satisfactory 

because  air  traffic  densities  are  lower  than  in  simulation  exercises.  This 

is  true  even  of  exercises  such  as  Red  Flag.  The  second  form  of  the 

problem  is  that  two-sided  war  gaming  is  not  possible  except  on  a  limited 

basis.  Isolated  intercept  exercises  ranging  from  one  interceptor  vs  one 

hostile  (1  v  1)  bump-heads  through  perhaps  four  interceptors  vs  four 

hostiles  (4  v  4)  are  possible  in  most  systems,  but  large-scale  exercises 

provide  essentially  no  interactive  play.  Except  for  IPS  tracks  (for  example, 

T-4  tracks)  which  are  interactive,  large-scale  exercises  are  largely 

2 

predetermined  and  unfold  without  regard  to  the  actions  of  the  C  team. 
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The  problem  of  low  geopolitical  relevance  has  a  motivational  and  content- 

oriented  impact  on  the  effectiveness  of  simulation  exercises.  Students 

expressed  boredom  and  disappointment  with  training  scenarios  based  on 

prior  conflicts  (for  example,  the  Korean  conflict).  Instructors  also  were 

2 

concerned  about  the  value  of  training  C  teams  to  deal  with  yesterday's 
battles  instead  of  tomorrow's.  The  students  and  instructors  felt  that  it 
would  be  valuable  to  learn  the  names  and  relationship  of  landmarks,  as  well 
as  the  airspace  and  force  coordination  procedures,  in  a  variety  of  world 
regions.  This  concern  was  expressed  especially  by  AW  ACS  instructors 
and  team  members  because  of  the  mobility  and  worldwide  responsibilities 
of  their  system. 

AWACS  Simulation  Exercise  Program  Packages  (SEPPs)  are  intended 
to  fill  this  need.  SEPPs,  which  are  families  of  related  exercise  problems, 
unfortunately  are  largely  noninteractive  and  do  not  present  a  realistic  threat 
in  a  currently  relevant  geopolitical  context. 

Important  training  support  functions  are  outlined  in  Chapter  2.  Many  of 
the  instructors  we  interviewed  expressed  frustration  over  the  lack  of 
support  provided  by  existing  simulators.  Many  of  their  performance 
monitoring,  assessment,  and  feedback  functions  could  probably,  in  their 
opinion,  be  automated,  thus  freeing  them  to  perform  more  productive 
duties.  Instructors  would  also  value  the  capability  for  modifying  the  details 
of  the  models  and  data  base  comprising  exercise  driver  scenarios. 

The  sense  of  frustration  felt  by  instructors  and  students  was  compounded 
by  the  feeling  that  communication  channels  and  procedures  do  not  exist 
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for  translating  their  observations  and  recommendations  into  incremental 
improvements  in  existing  training  devices.  This  problem  is  discussed  in 
more  detail  in  Chapter  4. 

ASSESSMENT  OF  TEAM  AND  SYSTEM  PERFORMANCE 

2 

The  primary  responsibility  for  assessing  C  performance  in  terms  of  team 
readiness  or  system  effectiveness  rests  with  the  personnel  of  the  standard¬ 
ization/evaluation  (stan/eval)  program.  Team  readiness  is  difficult  to 
assess;  the  use  of  subjective  measurement  methods  is  complicated  by 
factors  relating  to  the  conditions  under  which  performance  is  assessed. 

The  attempt  to  be  objective  in  measuring  system  effectiveness  is  subject 
to  influences  unrelated  to  effectiveness,  for  example,  mission  difficulty  or 
political  constraints. 

Standardization/ Evaluation  Program 

2 

The  purpose  of  the  C  stan/eval  program  is  to  ensure  that  the  performance 
of  individuals  and  teams  meets  the  standards  established  by  HQ  directives. 
Stan/eval  personnel  are  selected  through  a  nomination/approval  process 
involving  peer  judgment  of  the  individual's  performance  competency;  they 
receive  no  special  training  to  perform  their  duties.  Their  own  performance 
is  evaluated  by  stan/eval  personnel  from  higher-level  units  or  HQ. 

The  stan/eval  program  within  each  unit  is  organized  as  a  mini-version  of 
the  operational  sections  of  the  system.  That  is,  one  individual  is  assigned 
as  the  responsible  person  for  all  supervisors  and  technicians  within  a 
section. 
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The  stan/eval  program  consists  of  several  different  types  of  individual 
evaluation.  Phase  and  initial  evaluations  are  associated  with  approval  for 
OR  status.  Annual  and  spot  evaluations  monitor  stcills  maintenance.  Upgrade 
evaluations  are  the  basis  for  advancement  into  a  higher  skill  category.  All 
individual  evaluations  take  place  in  the  context  of  a  simulated  mission.  The 
rating  scale  method  is  used. 

Team  evaluation  is  carried  out  four  times  a  year.  The  vehicle  for  team 
evaluation  is  a  simulated  or  live  (as  available)  system  exercise  which  may 
vary  in  scope.  The  evaluation  is  oriented  towards  the  supervision  and 
performance  of  a  section,  for  example,  surveillance,  weapons,  etc.  The 
method  is  observation  and  the  evaluation  is  documented  in  an  after-action 
report  (AAR)  describing  the  positive  and  negative  aspects  of  team  per¬ 
formance.  Feedback  to  the  team  is  delivered  during  a  mission/exercise 
debriefing  and  through  the  AAR. 

The  AAR  provides  a  data  base  (organizational  memory)  for  determining  the 
objectives  of  subsequent  exercises.  Furthermore,  the  AAR  must  be 
responded  to  with  recommendations  for  corrective  action.  It  was  our 
observation  that  the  AAR  is  not  used  to  its  fullest  advantage  as  a  mechanism 
for  enhancing  organizational  memory  with  regard  to  team  and  system 
effectiveness. 

Difficulties  in  Assessing  Team  Readiness 

The  assessment  of  team  readiness  is  complicated  by  three  factors.  First, 
the  lack  of  standardization  of  conditions  under  which  performance  is 
evaluated  is  a  problem.  The  evaluation  may  take  place  in  a  live  or 
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simulated  environment,  and  difficulty  may  vary  widely.  Second,  the 
shortage  of  live  sorties  for  evaluation  (as  well  as  training)  means  that  the 
conditions  of  battle  are  only  marginally  approximated.  Third,  in  a  system 
like  AW  ACS  without  hard  crews,  the  team  evaluation  applies  on  a  one¬ 
time  basis  only.  There  is  thus  no  opportunity  for  the  particular  set  of 
individuals  to  learn  from  their  errors  made  in  working  cooperatively. 

Measures  of  System  Effectiveness 

Headquarters  personnel  have  responsibilities  which  prevent  them  from 
getting  involved  in  the  details  of  team  assessment.  They  monitor  individual 
progress  in  terms  of  any  failures  to  meet  training  requirements.  However, 
there  is  a  recognition  of  the  fact  that  even  if  all  members  of  a  team  are 
rated  as  operationally  ready,  it  doesn't  follow  that  the  system  will  be 
effective  in  meeting  mission  objectives.  In  order  to  get  a  grasp  of  this 
aspect  of  assessment,  mission  success  is  evaluated. 

2 

The  most  obvious  measure  of  success  of  TAC  systems  is  the  percentage 
of  hostile  tracks  identified  and  targets  intercepted/ killed.  This  measure  has 
been  used  in  SAGE  as  an  index  of  the  system's  state  of  readiness.  One  of 
our  respondents  pointed  out  a  shortcoming  of  this  measure.  Namely,  it 
doesen't  take  into  account  the  difficulty  of  the  conditions  under  which  it  was 
obtained.  Such  factors  as  the  number,  density,  and  speed  of  tracks  differ 
from  one  exercise  to  another.  Thus,  a  figure  of  95%  targets  intercepted 
is  misleading  without  some  indication  of  the  exercise  difficulty.  A  more 
valid  measure  could  be  obtained  by  weighting  the  score  by  a  difficulty 
factor,  much  the  same  as  is  done  in  Olympic  diving.  The  final  score  for 
individuals  or  teams  depends  upon  the  difficulty  of  the  dive  (exercise)  and  a 
rating  of  how  well  it  was  performed. 
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Another  method  of  assessing  system  effectiveness  is  buiit  into  large-scaie 
exercises.  Repeated  probiems  in  attaining  mission  objectives,  for  example, 
linking  AWACS  to  Navy  units  via  TADL-A,  come  to  the  attention  of  HQ 
personnel.  Such  problems  become  the  focus  of  exercise  planning  conferences.  * 
Thus,  the  determination  of  exercise  objectives  can  be  sensitive  to  issues  of 
system  effectiveness. 

\ 

MANAGEMENT  ISSUES 

Training  program  success,  in  addition  to  being  determined  by  methods  of 

training  and  performance  assessment,  is  heavily  influenced  by  management 

philosophy  and  policies.  Management  issues  are  present  on  two  different  ^ 

levels.  At  a  general  level  there  are  considerations  related  to  TAC2 

career  fields;  the  central  issues  concern  recruiting  and  selection, 

promotion  opportunities,  job  satisfaction,  and  the  cumulative  impact  of  all 

of  these  on  motivation.  There  are  also  factors  which  impact  specific 

training  programs.  These  include  1)  manager  expertise  and  knowledge, 

and  2)  management  of  training  for  new  systems. 
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Career  Field  Issues 


TAC  career  fields  are  not  among  the  more  glamorous  or  desirable  fields 
in  the  Air  Force.  At  the  same  time  there  is  currently  a  critical  need  for 
personnel  to  become  weapons  controllers  and  aerospace  warning  and  control 
technicians.  The  career  field's  problems  stem  from  two  sources:  1)  lack 

t 

of  visibility,  which  makes  it  difficult  to  attract  volunteers:  2)  undesirable 
characteristics  of  the  duty  and  promotion  possibilities,  which  make  it 
difficult  to  retain  quality  people. 

The  numbers  of  controllers  needed  exceed  the  supply  of  available  people 
^  and  the  manning  of  AWACS,  making  a  heavy  drain  on  the  current  complement 

of  controllers.  There  are  not  entry  qualificiations  for  this  field  at 
present.  Imposing  selection  criteria  would  reduce  still  further  the  avail- 
«  able  manpower  pool. 

The  stereotypical  job  in  the  Air  Force  is  that  of  flying  airplanes,  which  is 

unfortunate  because  there  are  many  other  technical  specialties  and  careers. 

2 

Of  the  many  necessary  support  functions,  TAC  is  probably  the  least  well 

«  understood  by  recruiters,  enlistees,  and  officer  candidates  alike,  if  they 

are  even  aware  of  them.  Consequently,  the  individuals  who  are  selected 
2 

for  TAC  career  fields  are  infrequently  volunteers  and  often  have  no  idea 
what  the  jobs  involve.  Our  instructor  respondents  reported  that  many 
student  controllers  arrive  at  Tyndall  without  having  had  any  orientation  to 
the  career  field.  Despite  this  and  any  disillusionment  it  may  cause,  the 
motivation  of  students  in  initial  training  is  usually  high,  although  student 
weapons  controllers  may  be  lacking  in  certain  prerequisite  skill,  for 
example,  mathematical  or  communication  (English),  because  of  the  absence 
of  selection  criteria. 
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Given  the  critical  need  for  C  personnel,  one  might  expect  that  there  would 
be  ample  promotion  opportunities  in  the  career  fields.  Unfortunately  for 
the  weapons  controller,  this  is  not  the  case.  The  number  of  17xx  personnel 
who  may  hold  rank  of  LTC  and  above  is  actually  quite  limited.  Pilots  do 
tend  to  dominate  in  this  aspect  of  career  advancement.  Pilots  receive 
flight  pay  but  AWACS  personnel,  who  spend  the  same  amount  of  time  in 
flight  as  their  pilots,  do  not  receive  flight  pay.  Our  17xx  respondents  were 
very  aware  of  this  and  were  outspoken  about  the  dissatisfaction  that  it 
causes. 

2 

TAC  duty  possesses  several  characteristics  which  make  it  undesirable 
according  to  our  respondents.  Periodically,  duty  assignments  involve 
serving  in  remote  locations,  for  example,  the  Aleutian  Islands.  Remotes, 
as  these  assignments  are  called,  pose  a  hardship,  especially  for  married 
personnel.  Although  remotes  are  viewed  negatively,  they  would  not  cause 
significant  problems  in  and  of  themselves.  Command  and  control  jobs  in 
TAC  in  peacetime  lack  relevance.  This  is  especially  true  for  TACS  and 
AWACS,  which  have  no  peacetime,  that  is,  non-tactical,  mission.  Live 
training  exercises  are  too  infrequent  to  have  a  positive  influence.  The 
primary  mission  of  SAGE/JSS  has  shifted  from  air  defense  to  air  sovereignty, 
so  that  the  day-to-day  activity  has  meaning.  However,  as  our  respondents 
reported,  even  in  this  context  the  jobs  of  surveillance  personnel  are 
tedious  and  the  weapons  section  may  go  days  waiting  for  a  low-frequency 
event,  that  is,  a  track  with  unknown  identity. 

2 

The  lack  of  promotion  opportunities  and  the  negative  features  of  TAC  duty 

2 

have  combined  to  significantly  demotivate  TAC  personnel.  This  has  led 
to  a  low  retention  rate  in  these  critical  career  fields. 


Program-Specific  Issues 


Training  program  managers  in  TAC  have  responsibilities  which  their 

previous  experience  may  not  fully  qualify  them  to  discharge.  Managers 

are  selected  not  according  to  demonstrated  ability  to  control  the  conduct  of 

training,  but  rather  according  to  rank  and  general  management  ability. 

Our  respondents  indicated  that  the  biggest  problem  was  a  lack  of  awareness 
o 

of  the  overall  TAC  training  pipeline  and  the  skill  levels  trained/achieved 
in  each  course.  It  was  difficult  to  assess  the  magnitude  of  this  problem,  but 
because  it  surfaced  at  TAC  HQ  its  importance  should  not  be  overlooked. 

A  second  issue  concerns  the  training  implemented  for  newly-acquired 
systems.  AWACS  training  problems  are  traceable  to  a  management 
philosophy  that  places  a  premium  on  successful  testing  during  acquisition. 

To  achieve  this  success,  training  decisions  are  made  which  in  the  long 
run  can  be  detrimental.  Two  examples  illustrate  the  problem. 

As  noted  previously,  personnel  for  the  JSS  will  be  crosstrained  from  SAGE. 
Consequently,  the  training  program  which  is  defined  and  developed  is  not 
designed  to  train  inexperienced  personnel.  There  is  no  problem  while  the 
SAGE  experienced  manpower  pool  lasts.  However,  as  soon  as  personnel 
turnover  requires  training  of  inexperienced  controllers  and  technicians, 
a  tremendous  strain  could  be  placed  upon  the  training  managers  at  JSS 
regional  sites.  At  that  point,  either  the  ATC  and  TAC  APQ  fundamentals 
courses  must  be  modified,  or  the  operations  (cross)  training  program  will 
have  to  be  expanded  to  include  a  front-end  initial  transition  course.  In 
this  instance,  it  is  likely  that  the  AWCT  and  APQ  courses  will  have  already 
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been  modified  to  prepare  controllers  and  technicians  for  JSS  duty.  The 
JSS  ROCC  Systems  Support  Facility  at  Tyndall  is  planned  for  use  as  a 
training  device. 

Despite  the  potential  problem  in  transitioning  the  JSS  program  from  cross¬ 
training  to  replacement  training,  there  is  a  plan  for  accomplishing  the 
transition  smoothly.  The  reason  for  discussing  this  issue  is  that  in  a 
historical  perspective  this  is  a  possible  explanation  for  how  the  AWACS 
training  program  got  into  trouble.  To  ensure  successful  development  and 
operational  tests  when  the  system  was  being  acquired,  the  training  was 
oriented  toward  experienced  individuals.  However,  there  was  no  existing 
manpower  pool  to  draw  on  since  AW  ACS  was  not  replacing  an  existing 
system.  Thus,  when  the  system  was  deployed  there  was  a  training  vacuum. 
The  result  has  been  implementation  problems  in  the  AWACS  training 
program.  The  ongoing  course  definition  and  development  would  have  been 
accomplished  much  earlier  if  the  new  system  "crosstraining  philosophy" 
had  not  dominated  the  training  pianning. 


CHAPTER  IV 


STATUS  OF  AFC2T2  PROGRAMS: 
EVALUATION  AND  MODIFICATION 


OVERVIEW 

Comprehensive  training  programs  have  procedures  for  evaluating  the 

effectiveness  of  training  and  modifying  training  in  response  to  discrepancies. 

Such  procedures  can  extend  to  the  evaluation  and  modification  of  training 

2 

devices.  In  this  chapter  the  procedures  for  evaluating  and  modifying  C 
training  programs  are  described,  analyzed,  and  themselves  evaluated. 

The  discussion  is  organized  around  three  topics: 

•  Program  evaluation  procedures 

•  Program  modific  tion  procedures 

•  Simulation/ simulator  evaluation  and  modification 
A  preview  of  the  findings  follows. 

Evaluation  is  a  key  step  in  the  ISD  process.  It  may  be  based  on  the 

trainees'  attainment  of  training  objectives  (internal  evaluation)  or  on  job 

2 

performance  of  recent  graduates  (external  evaluation).  In  C  training 
programs  both  types  of  evaluation  take  place.  Internal  evaluation  tends  to 
be  formal,  especially  in  initial  training  programs,  but  is  hampered  by 
incomplete  specification  of  performance  standards.  The  standardization/ 
evaluation  (stan/eval)  procedures  which  are  applied  in  operations  training 
programs  are  a  mixture  of  formal  and  informal.  The  procedures  are 
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described  and  on  balance  the  stan/eval  program  is  judged  a  strength;  but 
several  areas  for  improvement  are  noted.  External  evaluation  is  very 
informal  if  it  occurs  at  all  and  is  hampered  by  a  lack  of  objective  conditions 
and  criteria  for  job  performance,  which  leads  to  misunderstandings  between 
ATC  and  TAC  personnel.  An  example  arises  out  of  the  nature  of  the  inter¬ 
cepts  an  air  weapons  controller  is  trained  on  in  the  Fundamentals  Course. 
These  simulated  intercepts  are  "canned,”  that  is,  predictable  and 
specifiable.  Students  can  control  successfully  under  these  conditions,  but 
initial  transistion  training,  for  example,  at  the  607  TCTS,  is  based  on 
enetering  students  being  able  to  control  under  less  predictable  and 
specifiable  conditions.  Evaluation  of  large-scale  exercise  programs  is 
based  on  professional  judgment  and  some  of  the  potential  problems  with 
this  are  discussed. 

Programs  may  be  modified  using  ISD  procedures  as  long  as  the  entry 

point,  for  example,  job  task  listing  or  development  of  training  objectives, 

of  the  process  is  specified.  It  is  difficult  to  apply  ISD  to  modification  if 

either  the  program  was  not  developed  using  the  ISD  process,  or  the  process 

was  poorly  documented.  It  is  this  latter  problem  which  characterizes  the 
2 

status  of  C  training  program  modification.  This  problem  and  non-training 
factors,  such  as  resource  shortages  which  influence  program  modification, 
are  discussed  in  the  second  section. 

There  exists  no  systematic  procedure  for  identifying  simulator  deficiencies 

2 

and  potential  solutions.  AFC  simulator  design  does  not  lend  itself 

readily  to  modification  for  improving  capabilities.  In  general,  the 

2  2 

simulators  used  for  AFC  T  are  excellent  simulators,  but  marginal 
training  devices. 
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PROGRAM  EVALUATION  PROCEDURES 


There  is  no  comprehensive,  formal  set  of  program  evaluation  procedures 
used  for  both  internal  and  external  evaluation;  each  program  is  evaluated 
by  a  combination  of  different  procedures.  The  primary  procedures  in  use 
arel)  application  of  ISD,  2)  assessment  of  student  skill  level,  3)  achieve¬ 
ment  of  training  requirements,  4)  professional  judgment  of  the  state  of 
personnel  readiness,  and  5)  student  and  field  questionnaires.  Attrition 

rate,  which  might  be  a  useful  measure  of  training  effectiveness,  is  not 

2 

very  useful  in  the  TAC  environment  because  of  lack  of  selection  criteria 
for  weapons  controllers  and  because  the  rate  is  more  a  function  of  manage¬ 
ment  policy  and  pipeline  needs  than  student  success.  In  the  following 
subsections  each  evaluation  procedure  is  described  and  its  strengths  and 
weaknesses  are  discussed. 

Application  of  ISD 

Although  ISD  is  a  process  typically  associated  with  training  development, 
the  Air  Force  has  applied  ISD  to  existing  programs  in  order  to  determine 
if  they  provide  economical,  effective  instruction.  When  applied  with  this 
purpose,  ISD  should  begin  with  an  analysis  of  job  tasks.  Our  survey 
indicated  that  this  approach  to  program  evaluation  has  had  mixed  success. 

For  example,  the  Air  Weapons  Controller  Fundamentals  course  was  the 
subject  of  an  ISD  analysis  in  1978.  The  survey  results  indicated  that  this 
analysis,  rather  than  serving  as  the  basis  for  a  critical  comparison  of 
controller  job  skills  with  course  training  objectives,  was  used  merely  to 
justify  (validate)  the  then  ongoing  training.  On  the  other  hand,  the  ISD 
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analysis  performed  by  the  4444th  TACS/OTD  (Office  of  Training  Development) 
has  provided  a  meaningful  evaluation  of  the  TSQ-91  AWC  initial  transition 
training  program.  The  analysis  objectively  defined  AWC  job  tasks  and 
determined  training  reqflirements  by  comparing  entry-level  skills  with 
desired  on-the-job  skills.  An  interesting  byproduct  of  this  analysis  was 
the  discovery  that  the  entry-level  skills  of  ATC  Fundamental  course 
graduates  were  not  in  line  with  607th  TCTS  expectations.  The  TACS/OTD 
analysis  concluded  that  1)  student  proficiency  on  1  v  1  intercepts  was  not 
sufficient  for  transfer  to  the  TSQ-91  environment,  and  2)  students  were 
unprepared  to  perform  under  a  wide  variety  of  circumstances,  because 
Fundamentals  course  practice  conditions  were  arranged  so  that  intercept 
setup  and  positioning  were  predetermined. 

If  ISD  is  used  to  evaluate  a  training  program,  it  should  not  be  carried  out  by 

those  with  a  vested  interest  interest  in  the  outcome.  Otherwise,  the  results 

may  not  be  objective.  Unfortunately,  because  ISD  requires  subject  matter 

experts  (SMEs),  it  is  difficult  to  find  qualified  individuals  who  are  not 

associated  with  the  program  under  scrutiny  and  who  will  not  be  directly 

affected  by  the  results.  It  is  therefore  a  positive  sign  to  find  an  independent 

group,  the  4444th  TACS/OTD,  whose  sole  mission  is  training  development 

for  elements  of  the  Tactical  Air  Control  System.  The  TACS/OTD  serves 

as  a  model  for  overcoming  the  often  inherent  bias  when  ISD  is  used  to 

2  2 

evaluate  a  training  program;  this  is  an  AFC  T  strength. 

Assessment  of  Student/Team  Skills 


Within  the  ISD  framework,,  proficiency  assessment  is  the  standard  means 
of  evaluating  program  success.  If  students  meet  course  performance 
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standards  and  teams  are  judged  combat-ready,  then  it  is  commonly  inferred 

that  training  has  been  successful.  Of  course,  this  inference  assumes  that 

performance  standards  and  readiness  criteria  have  been  adequately  defined; 

this  is  not  always  the  case.  Course  performance  standards  are  not  always 

2 

objectively  defined  and  T  requirements  and  standards  are  not  formally 
developed.  Furthermore,  there  is  a  tendency  for  training  exercises  to 
become  ends  in  themselves  and  their  relationship  to  job  requirements  is 
rarely  well-defined. 

The  final  responsibility  for  assessing  proficiency  in  initial  and  operations 
training  programs  rests  with  stan/  eval  personnel.  As  discussed  in  Chapter 
II,  objective  written  tests  and  oral  exams  are  used  to  assess  knowledge 
proficiency  and  positional  performance  is  used  to  assess  skill  proficiency. 
Standards  are  rather  straightforward  in  the  area  of  knowledge,  for  example, 
attain  90%  correct,  but  not  so  in  skills.  In  particular,  the  criteria  for 
speed  and  accuracy  of  positional  performance  are  rarely  specified  con¬ 
cretely.  For  example,  our  survey  failed  to  discover  any  documented 
criteria  for  judging  the  positioning  accuracy  of  an  intercept  event.  Such 
standards  are  incorporated  informally  into  positional  evaluation  by  stan/ eval 
personnel  based  upon  their  expert  judgment.  The  expert  judgment  is  a 
strength,  but  the  informality  is  a  weakness. 

The  proficiency  levels  which  are  defined  in  conjunction  with  a  CTS  (course 
training  standard)  are,  therefore,  imprecise  and  open  to  interpretation. 

This  makes  it  difficult  to  evaluate  program  success  from  the  standpoint  of 
performance  on  the  job.  That  is,  TAC  and  ATC  experts  may  disagree  on 
whether  a  student  controller  has  achieved,  for  example,  level  2  skill 
proficiency,  because  of  this  imprecision;  or  if  they  do  agree,  they  may 
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discover  later  that  they  were  interpreting  the  description  of  proficiency 
differently.  Until  a  greater  effort  is  made  to  externalize  and  document 
performance  standards,  such  disagreements  will  continue  to  occur  (see 
previous  subsection).  The  result  is  that  in  the  case  of  skills  requiring 
technique,  proficiency  assessment  is  of  limited  validity  as  a  procedure 
for  evaluating  a  training  program. 

The  criteria  used  to  assess  team  readiness  in  system  or  large-scale 

2 

exercises  (informal  T  programs)  are  unspecified  and  the  assessment 

methods  are  relatively  informal.  The  stan/eval  personnel  have  no  documented 

guidelines  for  assessing  team  proficiency.  The  most  formal  aspect  of  the 

evaluation  procedure  is  the  generation  of  an  after-action  report  (AAR). 

These  reports,  however,  are  used  on  an  exercise-to-exercise  basis  rather 

than  serving  as  a  storehouse  of  exercise  program  evaluation  data.  Thus, 

the  assessment  of  team  readiness  as  it  is  currently  carried  out  does  not 

2 

provide  a  valid  or  efficient  procedure  for  evaluating  the  informal  T 
programs. 

I 

On  balance,  the  stan/eval  program  as  applied  to  individual  proficiency  is 
a  strength  without  which  quality  control  would  be  difficult.  Major 
discrepancies  are  prevented,  but  finer  discriminations  are  impossible.  The 
lack  of  objective,  behavior-referenced  criteria  for  assessing  individual 
team-oriented  skills  and  team  performance  limits  the  evaluative  power  of 
the  stan/eval  effort. 


Achievement  of  Training  Requirements 


Another  method  of  program  evaluation  consists  of  determining  whether 
training  requirements  have  been  achieved.  As  discussed  in  Chapter  II, 
training  requirements  for  operations  training  programs  are  defined  in 
terms  of  minimum  number  of  events/exercises  or  hours  per  topic  or 
activity.  The  attainment  of  minimum  training  experiences  is  clearly 
important  and  the  measures  produced  are  convenient  statistics  for  HQ 
personnel.  However,  it  has  not  been  established  as  to  whether  these  amounts 
of  events/activities  are  in  a  training- effective  quantity.  They  tend  to  be 
limited  in  number  and  type  by  resource  constraints  and  their  characteristics 
are  not  fully  responsive  to  training  objectives  for  job  performance  require¬ 
ments.  Although  the  minimums  have  been  established  and  endorsed  by 
experienced  personnel,  there  is  a  critical  lack  of  empirical  data  regarding 
the  meaning  of  attaining  the  requirements  in  terms  of  readiness  and  skills 
maintenance.  Therefore,  the  attainment  of  training  requirements  as 
presently  defined  does  not  constitute  a  necessarily  valid  procedure  for 
evaluating  operations  training  programs. 

Professional  Judgment  of  Readiness 

The  key  element  of  the  stan/eval  program  is  professional  judgment.  There 
are  also  less  formal  ways  in  which  professional  judgment  plays  a  role  in 
training  program  evaluation.  This  is  especially  evident  in  large-scale 
exercises,  which  are  based  on  developing  the  ability  in  personnel/systems 
to  meet  potential  threats,  for  example,  tactics  of  a  possible  adversary,  and 
in  informal  feedback  to  schools  from  the  field. 
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Professional  judgment  is  the  sole  basis  for  evaluating  whether  exercises 
like  Red  or  Blue  Flag  provide  sufficient  training  to  achieve  and  maintain 
combat  readiness.  The  evaluation  procedure  is  fairly  unstructured  and 
decentralized.  Any  individual  may  offer  a  critique  and  by  persuasion 
develop  a  consensus  that  training  must  be  modified  or  developed. 

This  procedure  is  essential  to  exercise  program  evaluation,  but  the  fact 
that  it  is  informal  means  that  it  is  based  on  the  appearance  of  a  vocal 
advocate.  That  is,  some  individual  must  feel  strongly  enough  about  the 
problems  to  go  on  record.  This  situation  is  subject  to  some  weaknesses. 

For  one  thing,  if  the  advocate  is  transferred  without  developing  a  consensus 
that  improvement  is  necessary,  the  motivation  for  modification  can  disappear 
For  another,  without  combat  experience  professional  judgment  becomes 
second-hand  or  speculation  based  on  intelligence.  Finally,  because  the 
procedure  is  informal,  the  evaluation  rationale  and  decision-process  may 
not  be  (well)  documented. 

Informal  feedback  from  the  field  to  the  schools  works  in  the  same  fashion 
and  is  subject  to  the  same  weaknesses.  However,  there  is  a  questionnaire 
technique  in  use  which  attempts  to  close  the  loop  between  the  operational 
and  institutional  settings. 

Student  and  Field  Questionnaires 


Questionnaires  are  employed  formally  in  program  evaluation  throughout 
2  2 

the  AFC  T  programs  surveys  for  two  purposes.  First,  the  students  are 
given  an  opportunity  to  critique  the  course  they  have  just  completed.  Second, 
the  opinions  of  operations  training  program  personnel  are  surveyed  by  ATC 
and  TAC  institutional  personnel. 


Institutional  training  program  managers  are  as  responsive  to  this  information 
as  they  can  be  within  the  limits  of  policy  and  available  resources.  But 
several  factors  work  against  the  utility  and  validity  of  the  information 
produced  via  the  questionnaires. 

Students  are  not  always  in  a  position  to  judge  whether  they  have  been  trained 
effectively  or  to  a  sufficient  level  of  proficiency.  Their  judgments  can  only 
be  based  on  whether  the  training  was  sufficient  for  them  to  pass  the 
knowledge  and  skill  proficiency  tests.  Because  these  tests  lack  formal 
assessment  of  team-oriented  skills  the  information  from  students  is  of 
limited  value.  Also,  students  tend  to  be  less  outspoken  and  critical  for 
fear  of  possible  adverse  effects  on  their  careers.  These  factors,  however, 
do  not  have  an  especially  noticeable  negative  impact.  The  solicitation 
of  student  opinions  is  a  valuable  management  tool  which  prevents  major 
discrepancies  in  training  administration. 

Feedback  from  the  field  regarding  the  qualifications  and  proficiency  of  the 
institution  graduates  is  useful  and  necessary  to  obtain.  However,  in  the 
weapons  controller  career  field  there  has  developed  a  "rift"  in  understanding 
between  the  field  and  school  personnel  because  of  the  way  in  which 
"controlling"  is  taught. 

In  the  school  environment  standardization  is  paramount.  Each  operational 
location  has  local  operating  procedures  and  requirements,  some  of  which 
represent  special  cases  and  therefore  deviate  from  standard  procedures 
taught  in  the  school.  Furthermore,  because  the  task  of  controlling  aircraft 
cannot  be  fully  proceduralized,  individual  biases  and  methods  come  into 
play.  Different  units  may  come  to  represent  different  schools  of  thought 
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on  controlling  techniques  based  upon  different  experiences.  The  institutional 
setting  and  the  field  setting  exemplify  different  schools  of  thought  in  this 
sense.  When  the  field  questionnaires  comment  that  the  school  doesn't  train 
the  proper  technique  or  criticizes  the  fact  that  the  operating  procedures 
taught  do  not  apply  to  their  location,  the  school  is  caught  in  a  bind.  They 
cannot  be  responsive  to  these  demands  because  they  must  turn  out  a 
standard  or  universally-assignable  product.  The  net  result  is  that  all 
feedback  from  the  field  is  taken  with  a  "grain  of  salt. "  The  problem  is 
that  institutional  training  programs  can  wind  up  operating  in  an  open- loop 
situation  and  take  on  a  separate  existence. 

PROGRAM  MODIFICATION  PROCEDURES 

Training  program  modification  is  necessary  when  evaluation  procedures 
provide  evidence  of  a  training  shortfall,  or  when  systems  or  missions  are 
modified.  The  survey  results  indicated  that  application  of  ISD  is  the 
primary  procedure  used  to  revise  training  programs;  systematic 
procedures  are  not  always  used  although  this  is  highly  desirable.  Occasionally, 
training  programs  are  modified  as  a  result  of  resource  limitations,  which 
may  include  changes  in  student  qualifications,  or  changing  management 
policy. 

Application  of  ISD 

2 

Where  ISD  is  used  as  a  tool  for  C  training  program  revision,  it  is  subject 
to  the  same  limitations  that  apply  to  its  use  for  definition  and  development. 
These  limitations  fall  into  two  categories:  ISD  structure  and  how  it  is 
applied. 


Foremost  among  the  structural  limitations  is  that  the  analysis  does  not 

adequately  address  the  definition  of  training  requirements  for  team-oriented, 

that  is,  cognitive,  skills.  Furthermore,  it  does  not  provide  for  the  design 
2 

of  T  exercises  of  simulated  combat  missions  in  terms  of  establishing 
training  objectives  and  exercise  characteristics  as  related  to  job  performance 
requirements.  Considering  that  these  two  problems  are,  according  to  the 
present  evaluation,  in  need  of  immediate  attention,  the  ISD  procedure 
itself  must  be  augmented,  modified  or  applied  in  novel  ways  before 
corrective  action  can  be  taken. 


In  the  method  of  application  category,  perhaps  the  biggest  limitation 

concerns  inadequate  documentation.  Any  training  program  modification  is 

much  easier  if  the  original  ISD  analysis  has  been  thoroughly  documented. 

2 

As  noted  earlier,  this  is  rarely  the  case  in  C  training  development. 
Without  such  documentation,  modification  may  require  that  earlier  thinking 
be  resurrected,  and  if  the  personnel  who  did  the  original  development  are 
gone,  the  process  becomes  time-consuming  and  is  wasteful.  An  audit  trail 
of  training  definition  and  development  decisions  is  essential  for  efficient 
program  modification. 

Non-Training  Factors 
2 

C  training  programs  are  occasionally  modified  as  a  result  of  a  shortage  of 
funds  or  manpower,  or  a  change  in  management  policy.  For  example,  the 
ATC  Fundamentals  Course  included  system  training  exercises  until  1975. 
They  were  discontinued  as  a  result  of  budget  cuts  and  the  end  of  the 
Vietnam  war.  At  the  same  time,  the  course  was  shortened  and  the  training 
standard  to  be  attained  was  lowered  from  control  of  two  interceptors 


137 


against  one  hostile  aircraft  (2  v  1)  to  one  interceptor  versus  one  hostile 
aircraft  (1  v  1).  The  consequence  of  this  change  of  standard  is  that  the 
operations  training  programs  or  initial  transition  training  either  must 
change  their  standards  or  increase  the  time  devoted  to  upgrade. 

Another  example  is  the  607  TCTS,  where  the  TSQ-91  initial  transition 
course  is  taught.  Until  recently,  the  607th  was  an  operational  unit  (CRC) 
with  a  full  complement  of  personnel;  during  that  period  students  were 
trained  in  team  skills  in  system  exercises.  When  the  unit's  status  changed 
from  operational  to  training  as  a  result  of  manpower  cuts,  this  team  skill 
training  could  no  longer  be  carried  out. 

Finally,  an  example  of  the  influence  of  management  policy  on  program 
modification  comes  from  the  ATC  Fundamentals  Course.  Recently,  ATC 
policy  fixing  the  length  of  a  training  day  at  eight  hours  has  been  enforced. 
Because  the  training  is  group  lock- step,  some  students  upon  completing  a 
lesson  cannot  continue  until  all  other  students  have  reached  the  same  point. 
Previously,  these  more  advanced  students  were  free  to  use  the  additional 
time  as  they  wished.  Now  they  must  be  present  for  eight  hours  and  the 
instructors  have  had  to  generate  academic  training  (which  they  feel  is 
meaningless)  to  fill  the  time.  Our  respondents  indicated  that  if  the  eight- 
hour  day  was  to  be  strictly  adhered  to,  the  course  should  be  self-paced  so 
that  better  use  could  be  made  of  the  time. 

SIMULATION/ SIMULATOR  EVALUATION  AND  MODIFICATION 

Evaluation  is  the  step  that  is  designed  to  close  the  loop  in  the  training  device 
development  cycle.  A  training  device  is  fielded  upon  completion  of  the 


definition  and  development  process.  While  the  system  is  in  actual  use,  a 
formal  procedure  should  be  followed  to  record,  consolidate,  and  analyze 
student  and  instructor  comments  about  design  deficiencies  and  potential 
solutions.  At  periodic  intervals,  the  feedback  should  result  in  incremental 
improvements  to  the  system.  This  process,  unfortunately,  rarely  occurs. 

The  T-2  and  T-4  systems  have  been  in  use  for  many  years,  and  their  major 
deficiencies  are  by  now  well-known  among  users.  The  capabilities  and 
algorithms  are  locked  so  tightly  within  the  electronics  of  the  systems,  however, 
that  a  major  effort  would  have  been  required  to  effect  significant  improve¬ 
ments.  Therefore,  the  strategy  that  was  adopted  was  to  use  the  system 
as  effectively  as  possible  until  it  could  be  replaced  entirely.  The  System 
Training  Exercise  Module  (STEM)  is  the  intended  replacement.  Until  the 
STEM  is  fielded,  the  training  programs  will  have  to  use  unimproved  T-2 
and  T-4  systems. 

2 

The  greater  use  of  software- controlled  C  systems  and  training  simulators 
suggests  that  improvements  may  be  easier  to  implement  than  was  the  case 
with  older  electronic  systems.  The  AWACS  simulator  demonstrates  that 
this  prediction  is  not  necessarily  valid,  however.  The  AWACS  simulator 
software  is  under  the  same  rigid  configuration  management  system  as 
is  the  operational  software.  The  extremely  conservative  configuration 
control  maintained  over  the  operational  airborne  software  is  appropriate 
because  of  the  strict  safety  and  reliability  standards  that  must  be  met, 
but  it  causes  serious  difficulties  for  the  training  system.  One  problem  is 
that,  except  for  IPS  consoles,  the  simulator  includes  essentially  no 
training  support  features.  Other  problems  are  that  the  SEPPs  provide 
inadequate  training  and  are  difficult  to  modify,  the  surveillance  team 
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and  command  staff  cannot  be  sufficiently  exercised,  support  personnel 
cannot  be  included  in  team  exercises,  and  relevant  variables  in  the  tactical 
environment  (ECM/ECCM)  cannot  be  exercised  well.  These  problems 
became  apparent  to  users  shortly  after  the  first  AWACS  simulator  was 
delivered.  Feedback  was  sent  to  the  designers,  but  it  had  no  significant 
impact.  The  second  AWACS  simulator,  delivered  in  1980,  is  essentially 
the  same  as  the  first.  The  Air  Force  now  owns  two  expensive  devices 
(over  $10  million  each)  that  are  exceptional  simulators  but  marginal 
training  devices. 
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CHAPTER  V 


CONCLUSIONS  AND  IDENTIFICATION  OF  HIGH-PAYOFF  ISSUES 

2 

Previous  chapters  of  this  volume  have  characterized  the  status  of  C 

training  programs.  The  methods  used  to  define,  develop,  implement,  and 
2 

evaluate  C  training  programs  have  be<"  Ascribed,  analyzed,  and 

2  2 

evaluated.  The  evaluation  permitted  idt..  .fication  of  AFC  T  strengths  and 
weaknesses. 

The  purpose  of  the  present  chapter  is  to  consolidate  the  weaknesses  into 
issue  areas  and  discuss  them.  The  issues  are  prioritized  according  to 
the  impact  their  resolution  is  anticipated  to  have  on  increased  training 
effectiveness  or  reduced  training  cost.  Subsequent  volumes  will  develop 
plans  and  recommendations  for  addressing  the  issues  through  1 )  a  program 
of  training  research  and  development,  2)  application  of  current  available 
training  technology,  and  3)  the  development  of  a  simulation  facility  for 
research. 

Many  of  the  issue  areas  identified  will  be  familiar  to  those  readers 

2 

acquainted  with  the  T  research  literature.  Progress  in  the  development 
of  an  accepted  team  performance  theory  has  been  slow.  If  a  mature 
scientific  field  is  characterized  by  having  a  shared  paradigm1  (Reference  19) 
then  the  science  of  team  performance  can  be  said  to  be  pre-paradigmatic. 

^'Universally  recognized  scientific  achievements  that  for  a  time  provide 
model  problems  and  solutions  to  a  community  of  practitioners" 

(Reference  19), 
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It  is  for  this  reason  that  a  number  of  the  issues  characterize  the  status  of 
2  2 

T  ,  whether  in  the  tactical  C  domain  or  not.  Paramount  among  these 

general  issues  is  the  absence  of  a  generally-accepted  conceptual  framework 

2 

for  the  definition  and  development  of  T  .  The  result  is  that  there  are  no 
2  2 

formal  T  programs  for  C  teams  and  systems  and  the  consequence  of  this 
is  that  team-oriented  skills  are  trained  in  an  unsystematic  manner,  at  best. 

In  order  to  focus  the  conclusions  and  a  discussion  of  the  issues,  the  areas 

2 

of  concern  have  been  categorized  according  to  whether  they  involve  T 

definition  and  development,  implementation,  program  evaluation  and 

modification,  or  personnel  policy  and  resource  constraints.  Table  12 

presents  the  issues  and  problem  areas.  Within  each  category,  the  issues 

are  listed  in  descending  order  of  resolution  priority.  The  categories 

themselves  are  also  in  priority  order  with  the  exception  of  the  constraints. 

These  are  presented  last  as  they  are,  in  general,  not  addressable  through 

research  and  development,  and  in  most  instances  are  part  of  the  fabric 
2  2 

of  .AFC  T  ,  which  must  be  taken  into  account  when  proposing  approaches 
to  resolving  any  issue. 

Issues  of  training  program  definition  and  development  have  the  highest 

priority,  because  their  resolution  is  fundamental  to  extended  progress  in 

2  2 

establishing  formal  T  programs  and  in  improving  T  efficiency  and 
effectiveness.  On  the  other  hand,  the  solution  of  more  specific  problems 
could  have  a  high  payoff  in  terms  of  enhancing  current  operations.  Also, 
a  higher  probability  of  success  can  be  expected  on  narrower,  presumably 
better- defined  problems.  The  discussion  of  each  issue  which  follows 
includes  a  summary  of  the  problem(s),  the  rationale  for  its  priority 
placement,  and  the  consequences  of  not  resolving  it. 


TABLE  12.  AFC  T  ISSUES  AND  PROBLEM  AREAS 


T2  DEFINITION  AND  DEVELOPMENT 

Six  issues  were  identified  as  important  in  the  category  of  training  program 

definition  and  development.  These  issues  have  the  highest  priority 

relative  to  the  other  training  program  areas  because  they  are  at  the  root  of 

2 

problems  and  weaknesses  found  throughout  T  programs.  The  issues  are 
ordered  in  descending  priority. 

Lack  of  a  Definitive  Framework  for  Identifying  and  Analyzing  Team  Aspects 
of  Operational  Systems  and  Defining  Team-Oriented  Task  Structure 

This  is  a  fundamental  issue  underlying  all  other  problems  and  weaknesses 
2  2 

in  C  T  .  If  this  issue  is  not  resolved  there  can  be  no  progress  except  of 

a  piecemeal  sort  on  the  narrowest  remaining  issues;  there  will  continue  to 

2  2  2 
be  lack  of 'emphasis  on  T  issues  during  C  system  acquisition  and  the  T 

that  is  developed  will  be  insufficient  to  accomplish  the  training  mission 

effectively. 

A  solution  to  this  issue  entails  the  development  of  a  conceptual  framework 

embracing  both  a  taxonomy  of  team-oriented  skills  and  a  methodology  for 

2 

considering  team  characteristics  in  C  system  design.  It  is  only  within 

2 

such  a  framework  that  the  most  critical  deficiency  in  T  ,  namely,  a  lack  of 
2 

specific  T  objectives  (Reference  20),  can  be  resolved. 

2 

The  characteristics  which  a  T  conceptual  framework  should  possess  are  as 
follows: 

2 

1.  A  classificatory  scheme  of  types  of  C  teams  according  to 
structure,  functions,  missions,  etc. 


144 


2.  An  identification  of  the  dimensions  of  the  process  of  team 
performance  and  variables  associated  with  task  output. 

3.  A  taxonomy  of  team-oriented  skills  along  with  a  task  analytic 
procedure  which  forces  attention  to  cognitive  skills  necessary  in 
cooperation,  coordination,  and  communication. 

*  2 

The  characteristics  of  a  C  system  design  framework  should  include: 

1.  Guidelines  for  determining  mission/task  functions  allocatable 
to  machines,  individuals,  and  teams. 

2 

2.  Guidelines  for  the  design  of  C  systems  which  efficiently  support 
the  functional  allocation  of  tasks. 

3.  Information  predictive  of  the  effects  of  task  or  position  automation 

2 

in  C  systems. 

9 

It  is  our  opinion  that  the  reason  for  lack  of  progress  in  the  area  of  team 

training  is  the  failure  to  deal  comprehensively  and  consensually  with  this 

fundamental  issue.  If  our  contention  that  team  performance  theory  is  in 

a  pre- paradigmatic  state  is  correct,  the  approach  to  resolving  this  issue 

%  should  consist  primarily  of  obtaining  descriptive  data  from  extensive 

2 

observations  of  C  team  performance.  The  observations  should  be  carried 
out  with  a  tentative  taxonomy  of  team  skills/behaviors  developed  in 

« 

accordance  with  the  cognitive  orientation  proposed  in  Chapter  in.  The  data 

base  so  acquired  will  provide  a  clear  picture  of  the  team  performance 

phenomena  which  must  be  explained.  The  initial  goals  of  this  effort  would 

2  2 

be  a  conceptual  framework  for  classifying  C  teams  and  a  model  of  C  team 
performance. 
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Lack  of  Objective  Criteria  and  Standards  for  Evaluating  Team-Oriented 
Skills,  Individual  and  Team  Readiness,  and  System  Effectiveness 

This  issue  is  quite  complex  since  it  covers  several  different  problems  of 

performance  evaluation.  There  exist  no  measurement  tools  to  1)  assess 

individual  team-oriented  skills  and  relate  their  level  to  some  standard  (of 

2 

readiness),  2)  assess  team  readiness,  or  3)  evaluate  C  system  effective¬ 
ness.  Each  of  these  problems  must  be  resolved,  possibly  with  different 
approaches. 

Resolving  the  problem  of  measuring  an  individual's  team-oriented  skill 
proficiency  requires  development  of  a  valid  task  taxonomy  and  this,  of 
course,  is  part  of  the  resolution  of  issue  number  1.  A  valid  taxonomy, 
however,  must  be  supplemented  with  performance  standards  which  reflect 
levels  of  proficiency  in  an  orderly  manner.  Such  standards  can  be  described 
a  priori  in  established  situations  where  the  responsibilities  and  operating 
procedures  are  defined.  But  the  absence  of  standards  is  part  of  the 
definition  of  emergent  situations;  there  may  be  more  than  one  acceptable 
solution  to  a  problem.  Another  way  of  looking  at  this  is  that  standards 
apply  well  to  performance  which  is  prescriptive  or  algorithmic  when 
particular  events  and  outputs  can  be  expected,  but  become  obscure  if 
performance  is  descriptive  or  heuristic  when  the  process  of  performance 
is  unpredictable.  This,  in  fact,  is  the  primary  reason  for  the  importance 
in  individual  evaluation  of  observers  who  are  experts  in  the  (standards  of 
the)  performance  under  observation.  What  is  needed  is  a  better  articulation 
of  the  standards  of  individual  team-oriented  skills,  one  which  takes  into 
account  the  emergent  nature  of  tactical  situations. 


Observation  of  the  process  of  performance  should  be  at  the  heart  of 
evaluating  team  readiness  and  system  effectiveness.  Readiness  cannot  be 
judged  solely  by  concrete  measures  of  number  of  exercises  or  activities 
engaged  in  per-unit  time,  or  percent  of  targets  detected/intercepted. 
Although  such  information  is  convenient  for  obtaining  summary  statistics 
for  use  by  higher  headquarters,  it  overlooks  critical  factors  such  as  the 
quality  or  difficulty  of  the  experiences  or  the  process  of  achieving  mission 
objectives.  Such  process  information  is  captured  to  some  degree  in  AARs, 
but  as  noted  below  these  are  not  used  to  full  advantage.  Part  of  the  reason 
they  are  not  used,  especially  for  program  evaluation  and  by  higher  head¬ 
quarters,  is  because  of  their  narrative  format. 

One  approach  to  resolving  this  performance  evaluation  problem  would  be 
through  the  application  of  multi-attribute  utility  theory  (MAUT).  MAUT 
provides  a  structure  for  analyzing  a  decision  situation  into  independent 
evaluative  dimensions  and  specifying  the  dimensional  attributes  character¬ 
istic  of  useful  outcomes  or  alternatives.  The  analysis  of  the  decision 
situation,  that  is,  the  expert's  basis  forjudging  readiness /effectiveness, 
should  provide  an  identification  of  the  dimensions  of  the  process  of  team 
performance.  Note  that  this  identification  is  a  characteristic  which  must 
be  possessed  by  the  framework  needed  for  the  resolution  of  issue  #1. 

The  suggested  approaches  to  resolving  the  performance  evaluation  problems 
focus  on  increasing  the  structure  and  orderliness,  thereby  the  validity,  of 
current  techniques.  The  Army  has  been  proceeding  along  these  same  lines 
with  the  ARTEP  (Army  Training  and  Evaluation  Program). 
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The  subject  matter  expert  is  still  the  only  valid  source  of  evaluative 
2  2 

information  in  C  T  .  The  primary  goal  of  measurement  development 
should  be  to  improve  the  expert's  tools. 

2 

Resolution  of  these  problems  is  a  prerequisite  to  conducting  useful  T 
research.  .Also,  without  resolution  it  is  difficult  to  ensure  standards 
for  quality  control  in  training  and  operational  readiness. 

Lack  of  Analytic  Techniques  and  Empirical  Data  for  Determining 
Institutional  and  Operational  T^  Requirements  and  Objectives 

The  successful  resolution  of  this  issue  depends  upon  the  resolution  of  issue 

#1.  Without  a  task  taxonomy  which  forces  attention  to  consideration  of 

team-oriented  tasks  producing  a  valid,  complete  task  list,  training 

requirements  and  objectives  cannot  be  defined.  This  is  true  whether  the 
2 

C  syste  n  in  question  is  in  development  or  deployed,  and  whether  the 
training  is  institutional  or  operational.  If  this  issue  is  not  resolved,  then 
training  cannot  be  improved  in  a  consistent,  verifiable  manner. 

The  approach  to  this  problem  should  be  through  an  extension  or  augmentation 
of  the  Interservice,  or  other  applicable,  ISD  procedure.  As  the  preeminent 
systems  approach  for  defining  and  developing  training,  ISD  must  be  used 
as  the  vehicle  for  such  an  effort.  It  is  our  opinion  that  the  problem  is 
tractable  within  the  ISD  framework,  given  some  revisions. 

A  revised  ISD  procedure  should  address,  among  other  things,  problems 

2  2 
within  C  system  acquisition.  Training  development  for  C  systems  which 

are  being  jointly  acquired  is  hampered  by  sometimes  conflicting  approaches 
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2 

to  and  constraints  on  training.  Training  developed  for  new  C  systems  is 
too  heavily  weighted  toward  experienced  individuals  thus  creating  undue 
pressure  on  institutional  and  operational  training  units  when  the  student 
pipeiine  begins  to  process  inexperienced  personnel.  Finally,  in  general, 
appropriately  derived  techniques  should  allow  training  developers  to  address 
issues  for  systems  which  are  unique  or  a  clear  departure  from  existing 
systems. 

Although  it  is  important  to  develop  analytic  techniques  for  establishing 
2  2 

initial  C  T  knowledge  and  skill  requirements,  it  is  equally  important  to 

2 

develop  a  data  base  regarding  C  skills  decay.  This  data  will  help  define 
refresher  training  (skills  maintenance)  requirements. 

The  current  practice  of  requiring  a  certain  number  of  system  exercises 
and  events,  or  hours  of  activities  per  month,  is  of  dubious  validity.  There 
are  no  empirical  data  establishing  the  relationship  between  performance 
frequency  and  combat  readiness,  and  such  data  are  needed.  The  relative 
priorities  of  this  and  the  preceding  issue  are  determined  by  the  fact  that 
measures  of  combat  readiness  are  a  prerequisite  to  skills  maintenance 
research. 

Lack  of  Comprehensive,  Systematic  Procedures  for  Defining 
Training  Objectives  for  Simulated  Combat  Missions 

Simulated  combat  missions  are  used  for  exercising  individual  positional 

and  team-oriented  skills.  This  technique  is  necessary  for  achieving  and 

2 

maintaining  combat  readiness.  Its  major  function  is  to  expose  C  teams  to 
the  quantity  and  quality  of  events  which  they  might  face  in  actual  combat. 
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The  events  themselves  may  be  live  or  simulated,  but  the  framework  within 
which  they  occur  is  entirely  fictional. 


The  specification  of  the  quantity  and  quality  of  these  events  is  the  subject 
of  this  issue.  These  events  must  be  responsive  to  the  training  objectives 
for  a  particular  exercise.  Working  group  conferences  meet  on  a  regular 
basis  to  plan  exercises.  However,  the  planning  procedure  is  rather  ad  hoc 
and  is  usually  constrained  by  limited  resources.  There  is  currently  no  set 
of  guidelines  for  defining  system  exercise  training  objectives,  especially 
when  the  system  mission  deals  with  emergent  situations. 

The  definition  of  training  objectives  for  system  exercises  is  complicated 
by  several  factors.  It  requires  the  expertise  in  military  tactics  and 
doctrine  plus  additional  knowledge  which  comes  primarily  with  combat 
experience.  Next  it  is  necessary  to  create  plausible,  meaningful  scenarios 
of  events  for  missions  which  must  capture  the  emergent  nature  of 
tactical  combat.  This  requires  a  great  deal  of  imagination  and  must  be 
tied  to  the  variability  of  mission  requirements  which  characterize  a 
worldwide  military  potential  (TACS  and  AWACS).  Furthermore,  the 
quality  and  quantity  of  events  must  be  tied  to  some  index  of  training  value, 
either  in  terms  of  team  skills,  mission  task  skills,  or  both. 

The  development  of  procedures  or  guidelines  for  defining  training  objectives 
for  simulated  combat  missions  should  be  approached  from  a  framework 
emphasizing  generic  team  skills  and  mission  tasks.  The  single  most 
important  quality  to  achieve  in  event  selection  is  variability.  It  is 
possible  that  the  planning  task  demands  exceed  man's  unaided  information 
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management  abilities.  One  approach  would,  therefore,  be  to  provide 
information  management  tools  for  planning  exercises.  Failure  to  resolve 
this  issue  means  that  system  exercises  will  continue  to  provide  unsystematic 
and,  perhaps,  insufficient  training  experiences. 

Inadequate  Planning  and  Analytic  Techniques  for  Defining 
Simulation  Fidelity  and  Functional  Requirements 

This  issue  reflects  the  fact  that  the  fidelity  and  capabilities  of  current 

simulations /simulators  are  more  related  to  the  level  of  technology  available 

or  acquisition  resource  limitations  than  to  what  might  be  required  to  support 

training.  Resolution  of  this  issue  entails  the  development  of  a  data  base 

relating  simulation  characteristics,  cost,  and  performance  benefit. 

Failure  to  resolve  this  issue  will  undermine  the  critical  and  expanding  role 

2  2 

that  simulation  must  play  in  C  T  . 

Some  of  the  specific  existing  problems  indicative  of  this  issue  are  discussed 

in  the  Implementation  section  of  this  chapter.  In  general,  the  approach  to 

developing  the  requisite  data  base  should  be  through  empirical  research 

conducted  by  using  a  facility  with  a  high  degree  of  functional  flexibility. 

2 

This  facility  should  be  capable  of  supporting  research  both  in  T  technology 
and  simulation  fidelity  and  capabilities.  Most  research  issues  will  fall 
in  neither  category  exclusively.  For  example,  the  use  of  voice  recognition/ 
synthesis  might  be  explored  as  a  training  technique  for  enhancing  the 
training  of  team  communication  skills,  perhaps  to  replace  or  augment 
intercept  pilot  simulators.  However,  the  required  simulation  fidelity  for 
voice  applications  is  also  a  question  for  research. 
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Failure  to  Define  and  Develop  Formal  Training 
for  C  ^  System  Supervisors 


This  issue  is  to  a  large  degree  a  matter  of  policy,  not  the  consequence  of 
a  lack  of  applicable  training  technology.  Strong,  competent  leaders  who 
thoroughly  understand  their  jobs  are  a  prerequisite  for  superior  team 
performance.  Yet  in  the  Air  Force  system- specific  training  for  supervisors/ 
managers  is  largely  on  the  job  (that  is,  during  simulated  mission)  and  thus 
is  informal,  unsystematic,  and  consequently  of  limited  value. 

The  training  technology  tools  exist  for  rectifying  this  situation.  Resolving 

the  issue  is  a  matter  of  modifying  policy  and  applying  existing  resources 

2 

to  developing  a  structured,  consistent  training  experience  for  C  system 
supervisors.  Such  a  training  experience,  for  example,  could  consist  of 
observational  learning  based  on  role-playing  of  a  prototypical  supervisor 
making  difficult  judgments. 

2 

Failure  to  resolve  this  issue  means  that  C  systems  will  not  have  super¬ 
vision  and  management  which  is  standardized  across  the  board. 

T2  IMPLEMENTATION 

Deficient  Simulator  Capabilities 

This  issue  is  clearly  related  to  the  weaknesses  in  simulation/ simulator 
definition  and  development.  It  is  treated  separately  because  the  problem 
areas  concern  deficiencies  in  current  training  simulators.  While  it  is 
true  that  these  deficiencies  are  symptoms  of  inadequacies  in  simulator 
development,  they  are  problem  areas  in  their  own  right. 
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Inadequate  simulation  of  ECM  and  sensor  management  has  presented 

2 

chronic  problems  in  all  C  system  simulations.  The  operational  community 
acknowledges  that  this  is  a  critical  deficiency.  However,  even  if  the 
8  requirements  for  adequate  ECM  simulation  were  known,  and  studies  should 

be  undertaken  to  determine  them,  there  would  still  remain  the  task  of 
implementing  them.  The  requirements  might  include  simulating  at  some 

4 

level  of  fidelity  the  output  characteristics  of  interference,  concurrent 
use  of  different  types  of  EW/ECM,  and  counter- countermeasures.  The 
task  of  writing  the  simulation  software  to  meet  these  requirements  would 
be  extremely  demanding  and  thus  costly.  But  the  cost  resulting  from  a 
lack  of  preparedness  to  wage  electronic  warfare  certainly  would  be  even 
,  higher. 

2 

The  lack  of  instructor  (simulator  operator)  support  capabilities  in  all  C 
»  trainers  results  from  the  use  of  operational  equipment  as  training  devices. 

It  is  less  expensive  to  buy  an  additional  copy  of  real  hardware  than  to 
modify  it  to  suit  the  training  mission.  But  the  long-term  consequences  of 
not  incorporating  such  support  may  be  more  costly.  The  time  available 
for  training  cannot  be  used  as  efficiently  or  effectively  as  it  could  be  if 
J  the  instructor  were  able  to  exercise  direct  control  over  the  simulation  and 

receive  performance  evaluation  information  (as  opposed  to  data  recording) 
automatically. 

4 

The  types  and  degree  of  support  capabilities  will  differ  according  to  whether 
the  application  is  institutional  or  operational  training.  For  example, 
training  controllers  at  independent  (institutional)  settings  might  require 
more  interactive,  real-time  control  by  the  instructor  than  training  a  team 
during  a  simulated  combat  mission  (operational  setting).  In  the  latter 
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instance,  simulation  of  an  intelligent  hostile  force  would  be  desirable.  The 
determination  of  what  kind  of  and  how  much  instructor/ simulator  operator 
support  to  provide  is  a  matter  for  research  and  development.  There  is  no 
doubt  that  improved  training  effectiveness  could  be  achieved  through 
enhancement  of  the  instructor's  abilities  to  direct  and  control  the  trainee's 
experience  and  provide  feedback  about  the  trainee's  performance. 

Currently  the  AWACS  simulator  provides  for  simultaneous  training  of  only 
a  portion  of  the  mission  crew,  primarily  the  surveillance  and  weapons 
control  sections.  There  is  no  provision  for  team  “xercises  which  include 
the  Airborne  Radar  Technician  (ART),  Communications  Technician  (CT), 
and  Radio  Operator  (RO),  nor  for  that  matter,  the  aircrew.  Coordination 
between  the  Air  Surveillance  Officer  (ASO)  and  ART  which  is  a  key  aspect 
of  mission  success  is  thus  trained  only  in  the  aircraft.  The  Computer 
Display  Maintenance  Technician  (CDMO)  console  is  provided  but  only  as 
necessary  to  initiate  the  system  simulation;  there  is,  for  example,  no 
fault  simulation  of  the  sort  that  would  require  coordination  among  various 
team  members  to  diagnose  (and  repair),  or  to  develop  procedures  for  work¬ 
ing  around  failures. 

The  result  of  incomplete  positional  simulation  is  that  training  time  on  the 
actual  equipment  in-flight  is  not  used  to  its  fullest  advantage.  Given  the 
cost  per  live  flying  AWACS  mission,  this  represents  a  waste  of  valuable 
resources.  The  drain  of  these  resources  will  continue  until  the  AWACS 
simulator  is  expanded  to  include  additional  crew  positions  and  a  fuller  set 
of  dynamic,  interactive  capabilities  which  promote  the  training  of  team- 
oriented  skills. 


154 


A  major  deficiency  in  simulation  is  the  lack  of  an  adequate  facility  for 

2 

system  and  superteam  exercises  in  which  C  personnel  can  be  systematically 
trained  and  evaluated  as  a  unit.  There  are  some  simulated  exercises  for 
units  and  two  or  more  units  but  their  major  weakness  is  the  emphasis  on 
operations  training;  the  training  of  the  battle  staff  as  integral  members  of 
the  team  is  haphazard  at  best. 

Blue  Flag  is  the  only  available  facility  and  it  has  many  limitations.  It  is 
remotely  located  for  most  units;  it  is  primarily  manual  in  operation;  the 
size  and  complexity  of  the  scenario  is  correspondingly  limited;  and  the 
collection  and  analysis  of  performance  data  is  limited,  subjective,  and 
informal. 

2 

There  is  a  need  for  a  well-designed  simulation  facility  for  C  teams  capable 
of  exercising  one  or  more  systems  of  the  size  of  a  CRC,  TACC,  AWACS, 
and  JSS  as  a  total  team.  Adequate  support  in  computers  and  personnel 
should  be  provided  to  operate  the  facility  and  to  meet  the  requirements  for 
training,  performance  evaluation,  recordkeeping,  and  war  gaming.  One 
central  facility  could  be  provided,  perhaps  supplementing  the  Blue  Flag 
facility.  Another  approach  is  to  provide  facilities  in  each  unit,  perhaps 
a  lesser  capability  but  more  readily  available  as  needed.  The  facilities 
in  two  or  more  units  might  be  interconnected  for  larger  exercises.  A 
third  option  is  to  permit  fielded  units  to  tap  into  a  large  central  facility 
by  telephone  lines  and  run  local  exercises  using  a  scenario  from  the 
central  facility. 


The  US  Navy  has  developed  two  simulations  which  are  centralized  facilities: 
Naval  War  Gaming  System  (NWGS)  and  Tactical  Advanced  Combat  Direction 
and  Electronic  Warfare  (TACDEW).  NWGS  is  a  surface  war  game  simulation 
to  exercise  the  command  staffs  of  a  naval  task  force.  The  people  are 
comparable  to  the  battlestaff,  TACC  personnel,  and  command  staff  of  a 
numbered  air  force.  Two-sided,  task  force  level  games  can  be  played  for 
several  days.  TACDEW  is  a  simulation  of  the  combat  information  center  for 
destroyers,  cruisers,  and  aircraft  carriers.  It  provides  high-fidelity 
simulation  of  command  operations  for  these  surface  ships. 

These  facilities  can  be  used  as  models  from  which  to  investigate  issues  of 
developmental  and  operating  costs,  design,  capacity,  and  use. 

Mismatch  Between  Entry-Level  Requirements  and 
Air  Weapons  Controller  Fundamentals  Course  Syllabus 

This  issue  must  be  considered  in  light  of  Air  Force  manpower  resources 
and  required  manning  levels.  At  present,  the  17xx  career  field  is 
critically  undermanned  and  this  situation  is  not  likely  to  improve.  Thus, 
although  psychometric  techniques  are  available  for  developing  selection 
criteria,  it  is  unclear  whether  such  criteria  could  be  applied  to  the  available 
officer  candidate  population.  As  long  as  demand  exceeds  supply,  the  Air 
Force  will  not  be  able  to  be  selective.  The  mismatch  between  what  is  taught 
and  what  is  required,  given  the  qualifications  of  the  students,  will  continue 
to  exist  until  corrective  action  is  taken. 


An  approach  to  resolving  this  inability  to  be  selective  for  the  AWC  career 
field  is  to  provide  supplemental  training  in  mathematical,  spatial  reasoning, 
and  English  (communication)  skills  at  the  AWC  Fundamentals  course.  These 
are  the  three  areas  which  our  respondents  indicated  were  characteristic 
weaknesses  of  AWC  Fundamentals  students  or  which  differentiate  between 
good  and  poor  controllers.  Failure  to  resolve  this  issue  places  a  burden 
on  all  subsequent  units  in  the  AWC  training  pipeline  and  has  obvious  conse- 
sequences  for  the  quality  of  personnel  in  the  AWC  career  field. 

Lack  of  Empirical  Data  Regarding  the  Optimal  Instructional 
Methods  and  Sequencing  for  Subteam,  Team,  and  Superteam  Training 

2 

A  systematic  training  methodology  for  T  does  not  exist.  Systematic 

approaches  to  training  presume  a  building-block  approach  in  which  components 

of  knowledge  and  skill  are  integrated  to  provide  behavioral  capabilities 

that  are  needed  on  the  job.  The  course  of  training  is  a  progression  through 

intermediate  behavioral  objectives  that  increasingly  approximate  terminal 

performance  objectives.  This  systematic  shaping,  however,  does  not 
2 

occur  in  T  .  There  are  two  reasons  for  this.  First,  team  knowledge 

and  skill  objectives  are  defined  only  in  broad,  general  terms.  The 

objectives  are  not  analytic  enough  to  support  sequencing  nor  selecting 

particular  training  methods  which  might  be  superior  to  others.  Second, 

existing  training  methods  might  not  support  the  needs.  In  fact,  there  is 

2 

a  need  for  data  regarding  the  success  of  various  methods  in  T  . 

The  resolution  of  this  issue  is  tied  directly  to  the  fundamental  issue  of 
definition  and  development,  as  discussed  above.  A  task  taxonomy  suited 
to  analyzing  team-oriented  training  requirements  and  objectives,  however. 


does  not  resolve  the  present  issue.  Rather,  in  conjunction  with  valid 

performance  measures,  it  provides  a  tool  for  gathering  data  regarding 

the  effectiveness  of  various  training  methods  in  meeting  the  objectives. 

2 

The  consequence  of  not  acquiring  the  data  base  is  that  T  will  continue  to 
be  unsystematic.  This  lack  of  structure  and  focus  undoubtedly  is  reflected 
in  inefficient  use  of  existing,  limited  resources  for  training.  Methods 
suitable  for  training  intermediate  behavioral  objectives  will  ensure  that 
personnel  subsequently  trained  in  system  exercises  of  varying  levels  of 
complexity  will  be  ready  to  gain  full  benefit  from  the  experience.  The  type 
and  degree  of  individual  pretraining  in  team-oriented  skills  to  incorporate 
in  intermediate  objectives  must  be  based  on  empirical  data.  The  same 
holds  true  for  the  use  of  hybrid  simulation  training  techniques,  where  the 
actions  of  some  team  members  are  modeled  and  the  trainee's  status  is 
that  of  a  man  in  the  loop. 

Lack  of  Training  for  Support  Personnel  Who  Simulate  Intercepter 
Pilots  During  Initial  and  Initial  Transition  Training 

This  issue  is  prominent  in  initial  and  initial  transition  training  programs, 
specifically  the  AWC  Fundamentals,  APQ,  and  TACS  407L  courses.  The 
individuals  who  simulate  intercepter  pilots  (interceptor  pilot  simulators-- 
IPPs)  receive  no  training  beyond  orientation  to  the  equipment  and  radio 
vocabulary,  pertinent  to  the  task  they  perform.  As  a  result,  the  training 
situation  for  AWCs  is  suboptima  1. 


A  related  problem  is  that  during  live  flying  in  these  same  courses,  the 
pilots  are  usually  students  themselves.  Because  their  skills  are  not 
necessarily  sharp,  the  training  experience  for  both  parties  can  leave 
much  to  be  desired.  For  example,  the  student  pilot  may  not  respond 
correctly  or  as  quickly  as  an  experienced  pilot  to  controller  instructions. 
Student  AWCs  mav  learn  to  compensate  for  these  errors  and  may  thus 
learn  techniques  inappropriate  to  communicating  with  experienced  pilots. 

In  either  of  these  two  circumstances,  under-trained  IPSs  or  controlling 
student  pilots,  the  problem  can  be  rectified  with  current  training  technology. 
A  training  course  could  certainly  be  developed  for  IPSs.  This  issue 

.  properly  falls,  however,  into  the  policy  area  and  as  such  is  not  directly  a 

candidate  for  training  research. 

’  On  the  other  hand,  this  problem  does  lend  itself  to  exploration  in  the  domain 

of  simulation.  In  particular,  it  may  be  possible  to  replace  the  IPS  with  a 
system  based  on  interactive  voice  technology.  The  Navy  has  had  some 
success  with  this  approach  in  training  air  controllers  for  precision  approach 
radar  tasks  and  is  in  the  process  of  applying  the  technology  to  training 
«  air  intercept  controllers,  the  Navy's  AWCs  (Reference  21). 

Failure  to  resolve  this  issue  means  that  early  training  experiences  for 
AWCs  will  continue  to  be  unrealistic.  It  is  possible  that  training  time  could 
be  shortened  if  optimal  use  were  made  of  IPSs  or  if  an  interactive  voice 
system  was  implemented,  given  that  the  concept  is  proved  valid. 
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Lack  of  AWACS-Oriented  Block  of  Instruction  in 
AWC  Fundamentals  of  APQ  Courses 

This  issue  is  primarily  a  matter  of  policy  and  presumably  will  be  resolved 
when  it  is  decided  which  half  of  the  initial  AWC  training  pipeline  will  feed 
AWACS.  The  consequence  of  not  resolving  this  issue  is  wasted  training  time 
for  AWACS-bound  AWC  students. 

Lack  of  Instruction  for  Supervisors,  Battle 
Staff  Personnel,  and  Decision-Makers 

Formal  training  for  these  categories  of  personnel  is  lacking  and  the  informal 
training  is  so  unsystematic  and  unstructured  that  its  effectiveness  and 
efficiency  is  vitiated.  This  deficiency  is  critical  because  these  persons 
are  critical  to  effective  team  functioning;  they  provide  leadership,  manage¬ 
ment  of  the  team's  resources,  and  quality  control  of  the  team's  processes. 
The  level  of  the  team's  proficiency  in  normal  operations  is  dependent  on 
these  roles  but  they  are  probably  more  critical  in  the  team's  capacity  to 
adjust  to  emergencies,  unanticipated  conditions,  high  workload,  and  stress. 

Some  formal  systematic  training  should  be  provided  for  these  positions  in 
2 

all  C  systems.  Unfortunately,  the  job  activities  are  complex  and  semi- 
structured  at  best.  Training  technology  has  neither  the  concepts,  the  tools, 
nor  the  analytical  techniques  to  develop  and  implement  this  type  of  training. 
At  least,  the  training  community  generally  believes  that  it  does  not.  There 
is  a  tendency  to  identify  this  domain  as  consisting  of  "decision  making"  and 
"emergent  situations"  and  throw  up  one's  hands  in  resignation  over  their 
intractability. 


160 


In  reality,  little  analysis  of  these  kinds  of  jobs  has  been  done.  Related 
analytical  work  has  been  done  by,  for  example,  Nickerson  and  Feehrer 
(Reference  22)  and  Ramsey  and  Atwood  (Reference  23) .  The  terminology 
and  concepts  are  available  to  provide  a  framework  to  describing  these  jobs. 
Furthermore,  the  behaviors  in  these  jobs  do  not  consist  wholly  of  unanalyzable 
decision-making  and  spur-of-the-moment  inventions  in  response  to  emergent 
events.  The  activities  are  largely  reducible  to  goals,  principles,  and 
rules,  and  the  use  of  skilled  routines  of  analysis,  interpretation,  and 
implementation. 

An  important  skill  is  monitoring  complex,  dynamic  events  and  information 
to  detect  events  and  conditions  that  require  action.  An  approach  to  training 
of  monitoring  skills  was  developed  by  McCutcheon  and  Brock  (Reference  24), 
and  Riley  and  McCutcheon  (Reference  25).  Their  objective  was  training 
watch  officers  on  naval  combat  ships  in  tasks  of  monitoring  and  evaluating. 
They  used  an  instructional  strategy  of  observational  learning  and  a  video¬ 
tape  medium.  Significant  improvements  in  performance  were  obtained  and 
the  method  is  neither  elaborate  nor  costly. 

T2  PROGRAM  EVALUATION  AND  MODIFICATION 

Two  of  the  issues  discussed  in  the  definition  development  areas  are  directly 
related  to  weaknesses  of  program  evaluation.  These  are  1)  the  lack  of 
objective  criteria  and  standards  for  evaluating  team-oriented  skills, 
individual  and  team  readiness,  and  system  effectiveness;  and  2)  the  lack 

of  analytic  techniques  and  empirical  data  for  determining  institutional 

2  2 
and  operational  T  requirements  and  objectives.  Valid  T  program 

development,  evaluation  and  modification  must  await  the  successful 
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resolution  of  these  two  issues.  However,  there  are  two  issues  specific 
to  program  evaluation  that  can  be  dealt  with  independently. 

Lack  of  Valid  Evaluative  Measures  for  Initial 
and  Initial  Transition  Training  Programs 

Current  institutional  (ATC  and  TAC)  training  program  evaluation  techniques 
include  the  use  of  questionnaires  for  two  purposes.  First,  the  students  are 
given  an  opportunity  to  critique  the  course  they  have  just  completed.  Second, 
operations  training  program  personnel  are  surveyed  by  ATC  and  TAC 
institutional  personnel  to  determine  if  student  skills  and  knowledge  meet 
expectations. 

Institutional  training  program  managers  are  as  responsive  to  this  information 
as  they  can  be  within  the  limits  of  policy  and  available  resources.  But 
several  factors  work  against  the  utility  and  validity  of  the  information 
produced  via  the  questionnaires.  The  students  are  not  always  in  a  position 
to  judge  whether  they  have  been  trained  effectively  or  to  a  sufficient  level 
of  proficiency.  Also,  they  tend  to  be  less  outspoken  and  critical  for  fear  of 
adverse  effects  on  their  record.  But  these  factors  do  not  have  an  especially 
noticeable  negative  impact. 

The  nature  of  feedback  from  the  field  is  more  severe  in  its  impact.  Each 
operational  location  has  local  operating  procedures,  some  of  which  represent 
special  cases  and  therefore  deviate  from  standard  procedures  taught  in  the 
institutional  environment.  Furthermore,  Decause  the  task  of  controlling 
aircraft  is  in  large  part  technique,  the  biases  of  individual  controllers  come 
into  play  in  any  criticism  that  is  offered.  Different  units  may  come  to 
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represent  different  schools  of  thought  on  techniques  for  controlling,  depending 
on  experiences.  AWACS,  SAGE  units,  and  TACS  units  are  subject  to  this 
latter  problem,  but  local  operating  procedures  only  impact  SAGE  and  TACS  units. 
The  institutional  environment  emphasizes  standardization  to  an  even  greater 
degree  than  the  operational  environment.  Consequently,  institutional  training 
cannot  be  responsive  to  these  demands  regarding  procedures  and  techniques. 

The  net  result  is  that  all  feedback  from  the  field  is  taken  with  a  grain  of 
salt.  The  danger  is  that  institutional  training  programs  can  wind  up 
operating  in  an  open-loop  situation  and  take  on  a  separate  existence. 

Several  approaches  could  be  taken  to  closing  the  loop  between  operational 
and  institutional  environments.  Better  survey  instruments  could  be 
constructed  with  the  problems  noted  above  in  mind.  Simply  raising  the  level 
of  awareness  of  training  program  managers  might  have  a  positive  effect. 

Effective  program  evaluation  techniques  need  to  be  implemented  with  a 
cooperative  rather  than  a  parochial  attitude. 

Incomplete  Use  of  Existing  Evaluative  Data  for 
System  Exercises  and  Simulated  Combat  Missions 

The  AAR  is  not  used  to  its  fullest  advantage  in  program  evaluation  and 
modification.  The  AAR  is  used  by  stan/eval  personnel  in  operational 
units  to  characterize  the  system  exercise  performance  of  the  unit.  An 
individual  AAR  may  have  an  impact  on  the  design  of  the  subsequent  exercise. 

In  any  event,  the  deficiencies  noted  in  the  report  are  the  subject  of  review 
and  corrective  action. 
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Despite  their  obvious  utility,  the  AARs  seem  to  serve  only  as  a  one-shot 
mechanism.  Rather  than  file  each  AAR  away  to  be  forgotten,  it  would  seem 
that  a  compilation  of  AARs  for  a  particular  system,  perhaps  with  some 
analysis  and  synthesis,  would  yield  a  data  base  descriptive  of  positive  and 
negative  aspects  of  performance  during  training  exercises.  At  the  very 
least  this  data  base  could  be  used  as  a  reference  for  new  team  members, 
especially  supervisors;  at  most  it  might  provide  a  useful  input  into 
developing  a  taxonomy  of  team- oriented  behaviors. 

If  this  issue  goes  unresolved,  an  opportunity  to  make  a  positive  contribution 
2 

to  T  would  be  lost.  There  should  be  a  high  payoff  for  a  small  investment. 

PERSONNEL  POLICY  AND  RESOURCE  CONSTRAINTS 

2 

Low  Retention  of  Experienced  C  System  Personnel 

Low  retention  rates  are  apparently  the  result  of  such  factors  as  low  pay, 
poor  work  conditions  or  assignments,  little  job  satisfaction,  and  the  lack 
of  clear  career  paths  and  attainable  objectives.  The  resolution  of  this 
issue  is  a  matter  of  policy  and  budget  rather  than  research  and  development, 
with  the  exception  that  attitude  surveys  might  be  useful  in  pinpointing 
problem  areas  and  providing  prescriptions  for  policy  decisions. 

The  consequences  of  not  resolving  this  issue  might  be  rather  severe, 
depending  on  how  low  the  retention  rate  actually  gets.  The  importance  of 
experienced  individuals  to  successful  institutional  and  operational  training 
and  to  the  achievement  and  maintenance  of  combat  readiness  cannot  be 
overestimated;  Definition  and  development  of  training  requires  subject-matter 
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expertise  in  C  systems  and  tasks;  the  primary  method  of  training  is  the 

master-apprentice  model;  program  evaluation  depends  upon  the  professional 

opinion  of  qualified  experts,  and  so  on.  If  there  were  a  cost-effective 

method  for  capturing  the  requisite  knowledge  and  skill  in  a  computer  (soft- 
2 

ware)  model  of  a  C  system  expert,  then  it  should  be  implemented.  Barring 
favorable  policy  and  budgetary  decisions,  this  might  be  the  only  feasible 
solution. 

2 

-  Shortage  of  Live  Flying  Intercept  Events  and  ECM  Activities  for  T 

It  is  often  claimed,  and  our  survey  was  no  exception,  that  there  is  no 
1  ,  substitute  for  the  experience  of  controlling  live  aircraft.  It  is  argued  that 

the  same  level  of  stress,  the  so-called  pucker  factor,  cannot  be  produced 
through  simulation.  (This  would  seem  to  be  an  empirical  issue,  however.  ) 

*  An  AWC  must  know  that  lives  are  on  the  line  to  find  out  if  s/he  can  indeed 

!  control  proficiently.  The  same  notion  axso  applies  to  teams,  in  the  sense 

of  survival  and  mission  success.  Obviously,  live  flying  is  necessary  for 
evaluative  purposes  as  well. 

2 

♦  Given  the  necessity  of  live  events  for  C  training  and  evaluation,  some 

j  problems  have  resulted  from  force  and  budget  reductions  in  recent  years. 

These  would  seem  to  be  constraints  that  must  be  lived  with.  Fuel  for 

» 

aircraft  will  not  get  cheaper.  The  prospect  of  increased  live  flying  to 
2  2 

support  C  T  is  unlikely.  What  can  be  done  ? 
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Ironically,  empirical  research  to  determine  the  relative  contributions  of 
live  and  simulated  events  to  training  and  proficiency  would  require  live 
resources  in  a  quantity  that  is  probably  unattainable.  In  addition,  the  cost 
of  obtaining  simulators  with  high  enough  fidelity  to  warrant  investigation  is 
prohibitive  given  present  resources  and  priorities.  One  approach  would  be 
to  justify  increases  in  fidelity  on  logical  argument  alone. 

Lack  of  Instructor  Training  and  Evaluation 
in  Operations  Training  Programs 

This  issue  is  a  matter  for  policy  rather  than  training  research  and  develop¬ 
ment,  because  current  technology  would  support  such  training  and  evaluation. 
.An  operational  command  could  receive  support  from  the  Air  Training 
Command  in  establishing  an  instructor  training  program.  Failure  to  resolve 
this  issue  means  that  operations  training  will  continue  to  be  less  effective* 
than  it  could  be  otherwise. 

Difficulties  Posed  in  Training  and  Evaluating 
Soft  as  Opposed  to  Hard  Teams 


There  are  elements  of  this  issue  which  are  appropriate  for  research  and 
development  but  the  basic  issue  of  team  stability  is  a  personnel  policy 
matter.  The  manpower  available  is  a  critical  factor  in  any  decision 
regarding  team  stability.  The  key  research  questions  which  must  be 
addressed  concern  1)  the  effect  on  team  skill  acquisition  of  constantly- 
changing  team  members,  and  2)  the  proper  approach  to  determining  soft 
team  combat  readiness  and  effectiveness  of  systems  manned  by  soft  teams. 
(For  a  similar  view  expressed  from  the  Navy  perspective,  see  Reference 


It  seems  evident  that,  early  in  training,  constant  switching  of  team  members 
would  tend  to  retard  skill  acquisition.  On  the  other  hand,  however,  in  the 
long  term  such  experience  may  lead  to  more  adaptable  team  members. 

Will  these  intuitions  be  proven  with  experimentation? 

.Although  it  is  possible  to  evaluate  a  team's  performance  given  proper 
criteria  and  standards,  it  is  unclear  what  the  meaning  of  such  an  evaluation 
would  be  if  the  team  were  composed  of  individuals  who  might  not  work 
together  again.  This  is  especially  disconcerting  if  one  is  attempting  to 
measure  and  generalize  about  readiness.  How  does  one  obtain  a  valid 
measurement? 

The  resolution  of  these  research  questions  is  imperative.  Without  answers, 
team  training  and  evaluation  must  continue  to  go  without  data  that  is  much- 
needed  to  aid  personnel  policy  decisions. 

2  2 

Inadequate  Understanding  of  C  T  Pipeline 
by  Training  Program  Managers 

This  final  issue  is  a  matter  of  personnel  policy  and  requirements;  it  does 
not  require  research  for  resolution. 

Current  training  technology  can  provide  suitable  training  packages  or 

criteria  for  selection  of  training  program  managers.  The  authority  to 

develop  such  information  must  first  be  given.  The  consequence  of  not 

resolving  this  issue  is  to  accept  the  status  quo  with  its  evident  program 

2 

management  weaknesses.  A  clearer  understanding  of  the  entire  C 
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training  system  by  individual  program  managers  will  equip  them  to  better 
evaluate,  modify,  and/or  develop  portions  of  the  course(s)  they  administer. 

Conclusion  t 


2  2 

The  state  of  C  T  in  Tactical  Air  Command  was  surveyed  and  summarized. 

Training  was  best  in  individual  initial  training  in  institutional  settings. 

Team  training  and  in-unit,  operational  training  was  more  rudimentary. 

ISD  procedures  are  used  for  individual  training  but  team  training  is 
unsystematic.  Team  training  is  usually  dominated  by  operational  activities 
that  are  feasible  in  terms  of  available  resources  rather  than  by  identified 
training  needs. 

* 

Systematic  definition  and  development  of  training  for  teams  and  team  skills 
is  virtually  nonexistent.  There  is  no  formal  training  for  supervisory  and  , 

battle  staff  personnel,  who  are  critical  to  the  effective  operation  of  teams. 

Simulation  is  used  for  systems  like  AW  ACS  and  CRC/CRP  but  the  number 

and  capability  of  these  devices  is  less  than  adequate.  The  simulation 
2  2 

facilities  in  AFC  T  are  significantly  behind  the  state  of  the  art.  Facilities 
which  can  provide  exercise  of  units  as  integrated  teams  and  joint  exercises 
of  two  or  more  units,  such  as  Blue  Flag,  have  serious  limitations. 

t 

Issues  and  problem  areas  were  identified  and  discussed.  They  will  be 
treated  further  in  other  volumes.  They  are  treated  in  two  groups:  those 
that  require  longer-term  research  and  those  that  can  be  addressed  and 
solved  in  the  short  term  using  currently  available  technology. 
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PREVIEW  OF  VOLUMES  II-V 


The  Identification  of  issues  and  problem  areas  completes  Volume  I  of  this 
report.  The  next  volume  (II)  presents  recommendations  for  a  research 
program  which  would  address  resolution  of  each  of  the  issues  in  a 
systematic  fashion.  Volume  III  discusses  those  aspects  of  issues  or 
problem  areas  that  can  be  addressed  through  the  immediate  application  of 
available  or  refineable  technology.  In  Volume  IV  the  functional  characteristics 
required  in  a  simulation  facility  to  support  the  recommended  research 
topics  are  described.  Finally,  Volume  V  presents  a  summary  of  the  entire 
research  effort. 


APPENDIX  A 

AFC2  SYSTEM  DESCRIPTIONS 
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AABNCP 


(Advanced  Airborne  Command  Post) 


Major  Command:  SAC 

Development  Agency:  ESD 

Deployment  Site(s):  Offutt  AFB,  Nebraska 

Current  Status: 

Description 

Function — AABNCP  has  been  undertaken  to  remedy  the  limitations  of  space, 
endurance,  and  communications  capability,  as  well  as  vulnerability  to 
nuclear  effects  associated  with  the  EC- 135  aircraft.  The  improvements 
include  increased  communications  capability,  enhanced  hardness  against 
electromagnetic  pulse,  increased  endurance,  and  a  larger  battiestaff  area. 
It  will  allow  antijam  secure  voice  and  data  communications  to  major  com¬ 
mands.  It  will  use  an  E4  (Boeing  747)  aircraft.  Phase  I  of  the  program  is 
complete;  three  D4  planes  were  equipped  with  the  C3  equipment  from 
existing  EC -135  aircraft.  The  present  phase  involves  the  development  and 
testing  of  advanced  C3  capability.  A  decision  will  then  be  made  on  the 
procurement  of  additional  C3  packages  and  aircraft.  Projected  fleet  size 
is  six  aircraft. 

Team  Composition — Command  Staff 

Related  Systems- - 

•  SACCS 

•  ALCC/ALCS 

•  AUXCP 


Contact — 


ABNCP 


(Airborne  Command  Post) 


i 


s. 


« 


4 


* 


Major  Command: 
Development  Agency: 
Deployment  Site(s): 
Current  Status: 

Description 


SAC  and  NCA 
ESD 

Offutt  AFB  (SAC  HQ) 
Operational 


Function — The  objective  is  to  provide  NCA  and  SAC  with  a  highly  survivable 
Ca  system  that  will  operate  reliably  during  all  phases  of  a  general  war. 

The  older  EC- 135  aircraft  now  employed  by  the  National  Emergency  Air¬ 
borne  Command  Post  (NEACP)  and  SAC  will  be  replaced  during  A  ABNCP 
with  E-4  aircraft. 

Team  Composition — Command  Staff 


Related  Systems — 

•  AUXCPs  (Auxiliary  Airborne  Command  Posts) — Augments 

ABNCP  with  relay  and  Airborne  Launch  Control  Systems  (ALCS) 
on  ground  alert  24  hours/day.  ALCS  aircraft  as  well  as  ABNCP 
and  AUXCPS  have  capability  of  launching  entire  Minuteman 
missile  force  if  they  lose  ground  control. 

Contact- - 


i 


« 


« 
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ALCS 


(Airborne  Launch  Control  System/ ALCC) 


Major  Command: 

Development  Agency: 
Deployment  Site(s): 
Current  Status: 


SAC/SAMSO  (Space  and  Missile  System 
Organization) 

ESD 

Offut  AFB 

Operational  and  developmental 


Description 


Function — ALCS  is  being  updated  to  receive  data  via  a  data  link  from 
Minuteman  silos  and  to  retarget  missiles  remotely  to  the  command  data 
buffer  capability  available  in  the  missile  launch  control  centers.  This 
capability  will  provide  a  highly  survivable  launch  platform  for  status 
and  retargeting.  This  improves  force  effectiveness  by  allowing  retargeting 
of  remaining  missiles  to  the  highest  priority  targets. 


Team  Composition — 

Related  Systems- - 

•  ABNCP 

•  AUXCP 


Contact- - 

ICMB  Program  Office 
Deputy  MGen  A.  W.  Hepfer 
(714)  382-6014 


* 


v 


> 


t 
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AWACS 


(E-3A  Airborne  Warning  and  Control  System) 


* 


4 


Major  Command: 
Development  Agency: 

Deployment  Site(s): 
Current  Status: 


TAC 

Air  Force  Systems  Command,  ESD, 
Hans  com  AFB 

Tinker  AFB,  Oklahoma  (one  of  several) 
Acquisition  and  operational 


Description 

3 

Function — Provides  survivable  airborne  air  surveillance  capability  and  C 
functions.  Its  distinguishing  technical  feature  is  the  capability  to  detect 
and  track  aircraft  operating  at  high  and  low  altitudes  over  both  land  and 
water.  Aircraft  may  deploy  throughout  the  US  and  overseas  to  provide 
surveillance,  warning,  and  control  in  a  wide  variety  of  peacetime  and  war¬ 
time  situations.  E-3A  is  a  Boeing  707  jetliner  equipped  with  an  advanced 
"look-down"  Westinghouse  radar  linked  to  sophisticated  data  processing 
equipment.  The  system  has  intrinsic,  indirect  self-defense  capabilities. 


Team  Composition — 

•  Officers  trained  at  Tyndall 

•  ES  trained  at  Keisler 


Related  Systems- - 

•  JTIDS  (tri-service  Joint  Tactical  Information  Distribution 
System)  involves  the  development  of  three  classes  of  terminal 
equipment.  Class  I  terminals  provide  a  jam- resistant  secure, 
high  data  rate  means  to  transmit  information  from  large  air¬ 
craft,  such  as  the  E-3A,  to  ground  and  airborne  forces.  JTIDS 
will  be  developed  in  the  AWACS  context.  JTIDS  links  Honeywell 
video  and  other  data  links  to  TACS  and  NATO.  Air  Defense 
Ground  Environment  system  provides  input  to  TACC/TACS. 

•  JSS/SAGE--In  wartime,  the  E-3A  will  deploy  to  ACCOM  bases 
and  interface  with  SAGE/JSS  personnel. 

Contact- - 

Deputy  B Gen  G.  W.  Rutter 
E-3A  System  Program  Office 
(617)  271-2711 
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BETA  (Battlefield  Exploitation  and  Target  Acquisition) 

Major  Command:  Joint  Army,  Air  Force,  and  DARPA 

Development  Agency: 

Deployment  Site(s): 

Current  Status:  Development 

Description 

Function- -A  joint  source  project  to  develop  and  demonstrate  a  test  bed 
consisting  of  three  mobile  fusion  centers  for  near-time  integration  of  data 
from  a  variety  of  surveillance  sensors.  The  goal  is  better  information  for 
maneuvering  and  targeting  purposes.  BETA  will  speed  up  the  targeting 
process  via  rapid  identification/ location  of  key  targets  and  the  distribution 
of  this  information  to  firepower  elements.  BETA  will  demonstrate  the 
feasibility  of  computer -assisted  decision-making.  It  will  enhance 
management  of  surveillance  and  reconnaissance  forces  and  assist  in  the 
prioritizing  of  specific  targets  and  effective  weapon  use. 

Team  Composition- - 

Related  Systems --OASIS:  BETA  is  a  smaller,  mobile  system  designed  to 
complement  OASIS  at  the  air  wing  level  for  target  acquisition.  BETA  will 
use  small  processors  for  target  allocation  whereas  OASIS  is  used  by  field 
commanders  for  allocation  of  a  large  number  of  forces. 

Contact -- 
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AD-A114  204  HONEYWELL  SYSTEMS  ANO  RESEARCH  CENTER  M 

TEAM  TRAINING  FOR  COMM ANO  ANO  CONTROL  S 
APR  82  0  R  BAUM.  J  A  MOORICK 

UNCLASSIFIED  afhrl- 

INNEAPOL IS  MN 
TSTEMSJ  STATUS. (U) 
F33615-79- 

TP-B2-7 

F/G 

C-002S 

NL 

- * 

>/Q 

1 

tm 

1 

■ _ _ _ 

BMEWS  TOR  UPGRADE  (Ballistic  Missile  Early  Warning  System  Tactical 

Operations  Room  Upgrade) 


Major  Command: 
Development  Agency: 
Deployment  Site(s): 


Current  Status: 


Successor  to  ADC 


I.  Greenland 

II.  Alaska 

III.  England 

Modification  and  analysis 


Description 

Function- -Upgrading  sites  is  expected  to  reduce  operations  staff  required 
by  half- -will  be  installing  new  display  consoles  and  automation  techniques. 

•  At  sites  I  and  II,  new  operator  consoles  will  replace 
obsolescent  consoles  to  improve  operating  efficiency 
and  reduce  number  of  personnel. 

•  At  site  III,  proposed  follow-on  will  provide  new  computers, 
improve  resolution  capability  on  radar  electronics,  and 
upgrade  the  Tactical  Operations  Room. 

Team  Composition- - 

Related  Systems- -NORAD 

Contact -- 
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JSS  (968  H  Joint  Surveillance  System)  Replaces  SAGE/ 301 C 

Major  Command:  ESD/FAC 

Development  Agency:  ESD 

Deployment  Site(s): 

Current  Status:  Validation/ acquisition 

Description 

Function-  -Program  has  been  established  to  acquire  and  deploy  a  peacetime 
air  surveillance  and  control  system  to  replace  the  SAGE  and  BUIC  (backup 
interceptors  command)  facilities  for  US  and  Canada.  The  strategy  is  to  get 
people  out  of  the  operating  loop.  For  Canada,  the  mission  is  expanded  to 
include  support  of  wartime  air  defense  functions  and  in  Alaska  the  mission 
includes  the  performance  of  tactical  air  control  functions.  In  wartime, 

JSS  would  be  supplemented  by  the  E-3A  AWACS. 

Team  Composition- - 

Related  Systems-  -  SAGE  /  B  UIC 

Contact -- 


i 


► 


* 


f 
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JTIDS  (Joint  Tactical  Information  Distribution  System  Mystic  Link) 

Major  Command:  Joint  Air  Force/Navy/Army  program 

Development  Agency:  ESD 

Deployment  Site(s): 

Current  Status:  Engineering  development 


Description 


Function- -A  key  digital  information  distribution  system  which  will  be 
secure,  jam-resistant,  and  survivable,  will  have  sufficient  capacity 
range  to  provide  current  combat  and  relative  position  information  to  all 
tactical  force  elements  on  a  near  real-time  basis.  May  be  first  field 
system  to  use  very -high -speed  integration  circuitry.  Will  connect 
AWACS,  ground  and  shipboard  C^  and  surveillance  centers,  and  combat 
and  support  aircraft.  May  be  adopted  by  NATO  allies  for  their  tactical 
data  needs. 


Team  Composition- - 


Related  Systems- -AWACS /TACS 


Contact--  Joint  Program  Office 

ESD,  Hanscom  AFB 

Col.  Brentnall,  Deputy  Manager  of  the  Joint 
Program  (617)  271-2714 
Lt.  Col.  Buckelew,  Deputy  Program  Manager 
for  Air  Force  (617)  271-2714 


MISSILE  LAUNCH 


Major  Command: 
Development  Agency: 
Deployment  Site(s): 
Current  Status: 


SAMSO/SAC 

ESD 

Vandenberg  AFB 
Operational 


Description 

Function- -Missile  launch  control  room,  Minuteman. 
Team  Composition- - 

Related  Systems -- 

•  ABNCP 

•  ALCC 


Contact- 


* 


4 
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NORAD  COC  (North  American  Air  Defense  Command  ' 

Combat  Operation  Center) 

Major  Command:  Successor  to  ADC 

Development  Agency: 

Deployment  Site(s):  Cheyenne,  MT;  Colorado  Springs,  CO 

Current  Status:  Acquisition,  deployment,  upgrade 

Description 

Function  -  -NORAD  COC  is  an  aerospace  surveillance  and  communications 
network  which  is  to  provide  unambiguous  and  early  warning  of  a  potential 
ballistic  missile  attack  and  provide  this  information  to  the  NCA,  Canadian 
Defense  Staff,  USCINCVEVR,  CINCLANT,  CENPAC,  and  CINCSAC. 

Team  Composition- - 

Related  Systems  — 

•  BMEWS 

•  Perimeter  Acquisition  Radars 

•  SLBM  Warning  System  (PAVE/PAWS) 

•  FPS-85  Radars 

•  Infrared  Launch  Detection  Satellites  (and  other  satellites) 

•  Integrated  Operational  Nuclear  Detection  System  (IONDS) 

•  Space  Detection  and  Tracking  System  (SPADATS) 


Contact-- 

I  - 
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OASIS  (2394  Operational  Application  of  Special  Intelligence  Systems) 

Major  Command:  USAFE 

Development  Agency: 

Deployment  Site(s):  Ramstein  AB,  W.  Germany 

Current  Status:  1978 — development  and  acquisition 

Description 

3 

Function  -  -Improvement  of  tactical  C  capabilities  through  the  application 
and  interfacing  of  appropriate  surveillance  and  special  intelligence  systems. 
Intended  as  an  automated  system  that  links  the  intelligence  world  and  the 
operations  world,  with  the  objective  of  enhancing  tactical  command  and 
control  activities  in  NATO.  OASIS  will  initially  use  off-the-shelf  hardware 
and  software,  originally  designed  for  peacetime  flight -following  computer 
systems,  and  apply  them  to  assist  decision-makers  in  handling  wartime 
resource  allocations.  OASIS  will  be  used  by  field  commanders  for  allocation 
of  large  numbers  of  forces. 

Team  Composition -- 

Related  Systems --BETA  is  designed  to  complement  OASIS  at  the  air  wing 
level  for  target  acquisition.  BETA  will  use  small  processors  for  target 
allocation. 

Contact  -- 


» 
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SACCS 


(Strategic  Air  Command  and  Control  System) 


Major  Command: 
Development  Agency: 
Deployment  Site(s): 
Current  Status: 

Description 

Function — 


Team  Composition  - 

Related  Systems -- 

•  ABNCP 

•  ALCC 

Contact-- 


SAC 

ESD 

Offut  AFB 
Operational 
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SATIN  IV 


(SAC  Automated  Total  Information  Network) 


Major  Command:  SAC 

Development  Agency: 

Deployment  Site(s): 

Current  Status:  Doubtful 

Description 

Function  -  -  Will  be  a  record  data  communications  system  for  SAC  and  a 
SAC  subsystem  of  WMCCS.  As  such  will  provide  secure,  two-way 
channels  of  communication  between  the  National  Military  Command  System 
(NMCS),  CINCSAC,  and  SAC  missile  and  aircraft  combat  crew  members. 

•  SACDIN  (SAC  Digital  Network) 

Formerly  SATIN  IV- -scaled  down  in  response  to  congressional 
demands  for  lower  costs.  SACDIN' s  purpose  is  to  furnish 
highly  responsible,  functionally  survivable,  and  secure 
communications  between  SACs  Commander-in-Chief,  the 
NCA,  and  SAC  missile  and  bomber / tanker  command  posts. 

Team  Composition-- 

Related  Systems -- 


t 


Contact-- 


SPADOC 


(Space  Defense  Operations  Center) 


Major  Command: 

Development  Agency: 

Deployment  Site(s): 

Current  Status:  Development 

Description 

Function --An  early  segment  of  the  developing  Space  Defense  Command  and 
Control  System,  SPADOC  is  scheduled  for  procurement  in  FY  1980.  In 
FY  1979  a  detailed  design  will  be  developed  for  SPADOC.  The  SDCCS, 
when  developed,  will  identify  outputs  from  the  current  SPACETRACK.  '  "It 
will  define  input  and  routing  for  maintaining  status  of  US  satellites, 
identification  of  forces  of  use  of  satellite,  and  define  command  and  control 
for  reporting  warnings  of  attack,  attack  impact  on  forces,  and  determining 
and  reporting  attack  verification. 

Team  Composition- - 

Related  Systems -- 

Contact--  ESD,  Hanscom  AFB 

A.  Salvucci,  Deputy  Program  Manager 
(617) 861-5432 


1 
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TACC 


(Tactical  Air  Control  Center) 


Major  Command:  TAC 

Development  Agency*. 

Deployment  of  Site(s):  At  least  Eglin  AFB  (in  support  of  Blue 

Flag  exercises) 

Current  Status:  Development  and  acquisition  production 

decision  planned  in  FY  78 


V 


Description 

Function-  -Operations  center  and  focal  point  for  all  air  activity  within  TACs. 
Coordinates  the  employment  of  tactical  air  effort  and  air  control  functions 
in  area  of  operations.  Has  four  groups:  1 )  Combat  Operations  Weapons 
Employment,  normally  referred  to  as  Current  Operations.  2)  Combat 
Operations  Plan,  normally  referred  to  as  Current  Plans,  which  centralize 
collection,  evaluation,  and  display  of  information  on  the  air  situation. 

These  two  operational  groups  have  parallel  groups  in  intelligence. 

3)  Fusion,  which  deals  in  real-time  data  collection  and  parallels  the 
combat  operations  weapons  employment.  4)  Intelligence,  which  parallels 
the  Combat  Operations  Plan. 

Team  Composition- -Command  Staff  Positions 

Related  Systems- -AWACS 

Contact--  Tactical  Air  Control  Center 

ESD,  Hans  com  AFB 

Col.  T.  Duff,  Deputy  Program  Manager 

(617)  861-4952 
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TACS 


(Tactical  Air  Control  System) 


Major  Command: 

Development  Agency: 

Deployment  Site(s): 

Current  Status: 

Description 

Function- -The  detection,  display,  and  control  equipment  of  the  485L  TACS 
provide  command  personnel  assistance  in  evaluating  the  current  situation 
and  directing  the  deployed  forces  in  accomplishing  the  counter -air  inter¬ 
diction,  close-air  support,  reconnaissance,  assault  airlift,  and  special 
missions.  Four  subsystems  support  the  Tactical  Air  Commander  and  his 
task  organizations: 

Subsystems 

1)  AC&W  (Aircraft  Control  and  Warning) 

--Detection,  identification,  and  control  of  all  aircraft  movement 
within  a  designated  area  of  responsibility. 

2)  DAS  (Direct  Air  Support  Subsystem) 

--Gives  fast  reaction  capability  to  satisfy  immediate  requests  from 
ground  forces  for  close -air  support,  reconnaissance,  and  assault 
airlift. 

Functions  carried  out  by: 

DASC  (Direct  Air  Support  Center) 

TACP  (Tactical  Air  Control  Parties) 

3)  ATC  (Air  Traffic  Control) 

--Promotes  the  safe,  orderly,  and  expeditious  flow  of  air  traffic, 
including  non-combat  aircraft--provides  maximum  airspace  use. 

Functions  carried  out  by: 

ATRC  (Air  Traffic  Regulation  Center) 

TATCF  (Terminal  Air  Traffic  Control  Facility) 
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4)  Communication  Subsystem 

--Provides  transfer  medium  for  much  of  the  information  flowing 
through  TACC  system. 

Team  Composition- - 

Related  Systems  -  -AWACS 

Contact -- 
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TAC  CIC 


(Tactical  Air  Command  Combat  Information  Center) 


Major  Command:  USAFE 

Development  Agency: 

Deployment  Site(s):  Europe 

Current  Status: 

Description 

Function  -- 


Team  Composition -- 
Related  Systems -- 
Contact -- 
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TACSI 


(Tactical  Air  Control  System  Improvements)  485L 
(407 -L  follow-on) 


Major  Command: 

Development  Agency: 

Deployment  Site(s):  Eglin  AFB,  Shaw  AFB,  Bergstrom  AFB 

Current  Status:  R&D  Acquisition 

Description 

Function --This  program  gives  TACS  increased  operational  capabilities 
needed  for  combat  C^  of  tactical  aerospace  operations.  Improvements 
consist  of  mobile  communications  and  electronic  systems  capable  of 
modular  world -wide  deployment  that  are  compatible  with  TACS  and 
interoperable  with  Army,  Navy,  and  Marine  Corps  tactical  data  systems. 

(F&S6.12)  * 

Team  Composition -- 

Related  Systems— TACS.  AWACS 

Contact-- 


# 
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TIPI 


(Tactical  Information  Processing  and  Interpretation  System) 


Major  Command:  TAC 

Development  Agency:  ESD 

Deployment  Site(s): 

Current  Status:  Development,  acquisition,  and  deployment 

Description 

Function- -TIPI  provides  the  capability  to  rapidly  exploit  the  information 
collected  by  reconnaissance  aircraft.  Its  objective  is  the  development  of 
air -transportable  facilities  that  provide  for  processing,  reduction,  and 
interpretation  of  aerial  reconnaissance  and  surveillance  products,  and  to 
collate  this  with  intelligence  from  other  sources. 

Terpes  is  one  segment  of  the  TIPI  system- -to  provide  ground  processing 
capability  for  determining  and  reporting  location  and  operating  parameters 
of  enemy  surface  emitters.  (TEREC  -*  TERPES  -*  TIPI) 

Team  Composition- - 

Related  Systems --AWACS/TACS 


Contact  -- 


Hanscom  AFB 

Col.  J.  Ostrominski,  Deputy  Program  Manager 
(617)861-4144 


TRACALS  (Traffic  Control  and  Landing  Systems)  404L 

Major  Command: 

Development  Agency:  ESD 

Deployment  Site(s): 

Current  Status:  Continuing  acquisition 

Description  » 

Function- -Fixed  and  mobile  ground  facilities  and  equipment,  with 
associated  avionics  to  update  the  USAF  air  traffic  control  function. 

Team  Composition -- 

Related  Systems- -TACS 

Contact--  1 


♦ 


I 
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TRI-TAC 


(Joint  Tactical  Communications) 


Major  Command:  TAC 

Development  Agency:  Ft.  Monmouth/ ESD 

Deployment  Site(s): 

Current  Status:  Development 

Description 

Function- -A  single  program  to  acquire  and  field  a  modern,  secure, 
multichannel  tactical  communication  system  for  all  services.  While 
achieving  interoperability,  the  goal  is  to  eliminate  duplication  and  reduce 
costs.  The  Director  of  TRI-TAC  is  the  architect,  the  systems  engineer, 
and  the  manager  of  the  program.  The  Air  Force  has  been  tasked  with 
five  TRI-TAC  development  programs. 

3  2 

For  the  purpose  of  C  T  only,  the  Tactical  Communications  Control 
Facility  (TCCF)  is  of  importance.  The  TCCF  will  perform  the  communi¬ 
cations  management,  near-real-time  control,  and  technical  performance 
monitoring  functions  for  a  communications  system. 

Team  Composition- - 

Related  Systems -- 

Contact  -- 


193 


-»'#i  urn  rii/*--  "ir  i  '•ajjj&mkm  i  "  *  **'•  ■**  •  *  ’*  ■-•«*****. 


USAFE  TFC/COIC 


(Tactical  Fusion  Center /Combat  Operations 
Information  Center) 


Major  Command: 
Development  Agency: 
Deployment  Site(s): 
Current  Status 

Description 

Function- - 


Team  Composition 
Related  Systems  -- 
Contact -- 


USAFE 

* 

Europe 


♦ 
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APPENDIX  B 


DATA  COLLECTION  METHODS  AND  DATA  COLLECTION 

INSTRUMENTS 


DATA  COLLECTION  METHODS 

In  order  to  achieve  the  objectives  of  this  research  effort  it  was  necessary 
to  carry  out  several  different  types  of  data  collection.  The  approach  was 
twofold:  1)  a  review  of  relevant  literature  and  technical  documentation 
and  2)  a  field  survey  consisting  of  interviews  of  training  personnel  and 
observations  of  training. 

Literature  and  Technical  Documentation  Review 


This  section  discusses  three  aspects  of  the  review.  A  major  initial  input 
to  this  effort  was  an  AFHRL  review  of  the  literature  on  team  training 
research  (Denson,  1980).  Other  relevant  literature,  for  example,  in  the 
area  of  C2  decision-making,  was  identified  during  the  course  of  this  effort 
and  was  reviewed.  Technical  documentation  concerning  Air  Force  C2 
systems  was  examined  to  provide  rationale  for  survey  site  selection,  as 
discussed  in  the  Field  Survey  section.  Also,  during  the  site  surveys  we 
gathered  documentation  on  training  programs  including  course  outlines 
and  materials,  training  development  information  and  evaluation  materials, 
and  on  the  system  planning,  development,  and  acquisition  process. 

2 

AFHRL  Literature  Review--This  T  literature  review  examines  definitional 
issues  and  those  related  to  characteristics  of  individuals,  tasks  and  teams, 
feedback  and  performance  objectives,  and  measurement.  Also  included  is 
the  applicability  of  instructional  system  development  (ISD)  techniques  to 
the  development  of  T2. 

The  review  served  to  focus  the  topics  of  data  collection.  The  issues  raised 
in  the  review  certainly  exist  to  one  degree  or  another  in  the  C2T2  area. 

In  Chapter  3  of  Volume  I  and  in  Volumes  II  and  III,  previous  research 
results  are  brought  to  bear  on  the  analysis  and  evaluation  of  C2T2  in  the 
Air  Force. 


o 

Additional  Literature --As  the  research  progressed  and  T  issues  suggested 
themselves,  additional  information  sources  were  sought.  Reports  unavailable 
for  the  AFHRL  review  were  identified  and  reviewed.  The  results  of  this 
activity  will  be  evident  throughout  this  report  as  the  literature  reviewed  is 
given  appropriate  reference. 

Technical  Documentation --This  documentation  falls  into  two  categories: 

1)  systems  and  2)  training  program  and  system  acquisition  materials. 

The  literature  on  C2  systems  was  reviewed  to  determine  system  nature, 
status,  and  deployment.  The  primary  source  for  such  information  was 
articles  in  various  Armed  Forces  and  electronics  magazines.  Selection 
of  sites  to  be  surveyed  was  based  in  part  upon  data  from  this  review. 

During  the  data  collection  effort,  we  obtained  numerous  documents  and 
materials  pertaining  to  various  aspects  of  Air  Force  training;  system  - 
specific  training  and  T2;  C2  systems  specification,  design,  and  acquisition; 
and  the  like.  This  data  has  been  thoroughly  integrated  into  Chapters  2-4 
of  Volume  I,  forming  a  portion  of  the  description  of  C2T2  status.  The 
material  is  referenced  as  appropriate  throughout  and  is  included  as 
appendixes  where  appropriate. 

Field  Survey 

There  are  three  aspects  of  the  field  survey  which  must  be  discussed: 
population,  methods,  and  site  visits. 

Survey  Population- -Preliminary  considerations  in  matching  methods  to 
objectives  involved  identifying  the  types  of  personnel  who  would  contribute 
to  the  research  data  base. 

Training  personnel  clearly  had  to  form  a  major  part  of  the  sample.  Design 
and  development,  implementation,  and  evaluation  are  the  important  stages 
in  a  training  program's  life  cycle.  Therefore,  personnel  involved  in  each 
of  these  stages  had  to  be  interviewed;  these  personnel  include  managers, 
training  technologists  and  instructional  system  designers,  instructors, 
evaluators,  and  students.  In  addition,  because  there  is  an  intimate 
relationship  between  C6  system  capabilities  and  training  requirements, 
the  C2  system  acquisition  process  had  to  be  examined  to  reveal  when  and 
where  training  considerations  impact  system  design.  Therefore,  C2 
system  acquisition  planners  had  to  be  interviewed. 
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Because  it  was  desired  to  informally  evaluate  T  program  success, 
operational  unit  supervisors  /commanders  also  had  to  be  interviewed  when 
possible.  It  was  expected  that  these  individuals  could  provide  us  with 
insight  as  to  the  suitability  and  effectiveness  of  earlier  training  and  any 
necessity  for  accomplishing  further  training.  By  way  of  preview,  in  all 
cases  the  operational  units  we  surveyed  were  at  once  users  and  trainers, 
so  this  approach  was  convenient  and  productive. 


In  order  to  make  recommendations  for  the  establishment  of  a  T  research 
and  development  (R&D)  program,  it  was  necessary  to  survey  future  C2 
system  capabilities.  A  meaningful  T2  R&D  program  is  dependent  upon  a 
valid  assessment  of  future  requirements,  which  in  turn  depends  upon 
future  C2  team  tasks  and  structure.  Achieving  this  objective  required  a 
survey  of  the  status  of  technology  development;  thus,  knowledgeable 
Air  Force  and  industry  personnel  were  interviewed. 


Survey  Methods --The  survey  requirements  were  met  through  interviews 
and  observations  (and  technical  document  data  collection,  as  noted  above). 
The  interviews  and  observations  were  guided  by  data  collection  instruments 
(DCIs)  developed  specifically  for  this  effort.  The  DCIs  were  developed  in 
a  three-step  process,  as  discussed  next. 


PCI  Development --The  first  step  in  DCX  development  was  to  clearly  define 
the  areas  of  emphasis  and  information  objectives  of  the  effort.  The  results 
of  this  step  were  the  specification  of  content  areas  and  questions  to  be  asked 
or  observations  to  be  made  within  each.  The  approach  was  to  develop 
interview  schedules  for  each  type  of  respondent  who  would  contribute  to  the 
data  base,  and  to  develop  a  general  observation  guide  which  would  be 
applicable  to  all  training  to  be  observed.  Five  DCIs  were  developed  for 
interviewing; 


•  Training  program  manager /developers 

•  Instructors 

•  Students  and  recent  graduates 

•  Operational  unit  supervisors  or  commanders 

•  Systems  Program  Office  staff 


One  observation  guide  was  developed.  The  DCIs  are  presented  at  the 
second  part  of  this  appendix. 


198 


4 


% 


t 


The  second  step  in  DCI  development  was  determination  of  a  preliminary 
format  and  contents  for  review  and  use  during  early  data  collection  efforts. 

A  heview  of  the  literature  on  interview  techniques  revealed  a  useful 
distinction  between  interview  schedules  and  guides  (Gorden,  1969).  A 
schedule  refers  to  a  highly  structured  and  detailed  format  in  which  all 
topic  areas  and  questions  to  be  asked  and  their  sequence  are  specified 
for  the  interviewer.  A  guide  refers  to  a  format  which  includes  the  topic 
areas  and  some  example  questions  or  probes.  There  is  no  order  of 
coverage  implied  and  the  guide  may  be  used  as  a  memory  aid  or  checklist. 
The  guide  format  was  judged  desirable  since  it  permits  a  more  natural  and 
freewheeling  interview  and  ensures  the  gathering  of  information  pertaining 
to  all  topic  areas,  but  does  not  stifle  spontaneity  or  in-depth  coverage  of 
particularly  productive  or  unanticipated  topics. 

The  guide  format  was  also  chosen  to  assist  data  gathering  during  the 
observation  of  training.  The  topics  selected  for  observation  specify  the 
characteristics  of  equipment,  training  programs /exercises  which  were  to 
be  observed  during  site  visits. 


Before  finalizing  the  preliminary  DCI  format  and  contents,  a  draft  version 
of  the  guide  for  interviewing  the  Training  Program  Managers  /Developers 
was  used  in  interviews  with  two  Honeywell  personnel.  These  personnel 
were  responsible  for  developing  the  individual  and  T^  for  the  Army's 
Stand-Off  Target  Acquisition  System  (SOT AS)  program.  This  approach 
permitted  the  interviewers,  before  going  into  the  field,  to  determine  a 
natural  order  of  topics  for  discussion,  and  the  utility  of  phrasing  questions 
in  certain  ways.  It  also  helped  to  sharpen  their  interview  skills.  Also, 
information  regarding  current  approaches  to  C^T^  (SOTAS  is  a  system) 
was  added  to  the  data  base. 


This  approach  was  not  possible  with  the  observation  guide,  since  the  SOTAS 
training  program  was  dormant  at  the  time.  However,  observation  by  its 
nature  lends  itself  to  change;  it  is  a  more  flexible  data  gathering  technique 
than  interviews,  so  the  risk  in  this  area  was  small. 

The  third  step  in  DCI  development  was  revision  of  the  instruments  based 
on  actual  use  in  the  field  and  review  with  AFHRL.  As  discussed  below, 
the  interview  format  underwent  some  changes  based  on  experience,  but 
the  content  of  the  interview  and  observation  guides  did  not  change. 
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PCI  Use --The  field  environment  is  not  as  controllable  as  the  confines  of  the 
laboratory.  The  DCIs  were  used  as  planned  when  such  use  met  the  objectives 
of  the  study.  However,  meeting  the  objectives  occasionally  called  for  plans 
to  be  modified.  The  approach  chosen  was  flexible  enough  to  permit  the 
necessary  modifications. 

The  plan  for  interviews  was  to  use  to  interviewers  with  one  respondent. 

The  interviewers  were  to  operate  as  a  team,  one  asking  questions,  the 
other  taking  notes.  Because  it  was  projected  that  the  interviews  would 
be  long,  on  the  order  of  two  to  four  hours,  it  was  planned  that  the  interviewers 
would  switch  roles  about  halfway  through  the  session.  In  addition,  in  order 
to  relieve  the  burden  on  the  note  taker,  it  was  decided  to  tape  record  the 
interviews,  not  for  the  purpose  of  transcription,  but  only  to  supplement  and 
clarify  notes.  Finally,  it  was  planned  to  interview  all  respondents 
individually,  with  the  exception  of  students  and  recent  graduates,  who  were 
to  be  interviewed  in  small  groups. 

Observation  of  training  and  exercises  was  to  be  non-instrusive  to  the  ongoing 
activity.  Each  training  program  or  exercise  observed  was  to  be  followed 
by  a  debriefing  of  a  representative  sample  of  the  personnel  involved.  The 
format  for  these  debriefings  was  planned  to  be  the  same  conversational, 
flexible  approach  adopted  for  the  interviews. 

The  interviews  proved  to  be  at  least  as  time-consuming  as  anticipated,  in 
fact,  more  time  than  planned  was  required  during  early  site  visits  and  so 
the  length  of  time  set  aside  for  each  trip  was  expanded  to  give  maximum 
flexibility.  The  two -interviewer  format  was  dropped  when  time  during  a 
site  visit  ran  short,  but  this  occurred  after  the  early  site  visits.  By  that 
time  the  interviewers  had  polished  their  skills  auid  become  conversant  in 
the  particulars  of  Air  Force  C^T^.  Also,  to  save  time,  non-student 
respondents,  for  example,  instructors,  were  occasionally  interviewed  in 
small  groups  averaging  three  to  five  individuals. 

Early  in  the  study  it  was  determined  that  the  tape  recorder  created  more 
problems  tham  it  solved.  Notes  and  follow-up  conversations  proved 
sufficient  to  gather  the  needed  data  so  dispensing  with  the  recorder  did 
not  affect  the  validity  of  the  effort.  The  problems  with  the  recorder 
included  possible  inhibition  of  conversation  auid  inconvenience  in  its  use. 

The  tapes  were  helpful  in  validating  the  general  interview  approach  but 
proved  difficult  to  use  as  a  memory  aid  for  specific  instances.  This  was 
because  the  specific  instances  in  question  had  to  be  retrieved  from  three 
or  more  hours  of  recording,  which  consumed  significantly  more  time  tham 
a  follow-up  phone  call. 


Another  modification  in  method  evolved  over  the  course  of  the  study.  The 
interviews  during  early  site  visits  were  conducted  by  following  the  guides 
very  closely.  At  any  particular  site,  however,  there  proved  to  be  topics 
the  content  of  which  remained  the  same  from  respondent  to  respondent. 

It  was  therefore  redundant  and  wasteful  of  time  to  pursue  certain  topics, 
for  example,  system  configuration,  with  more  than  one  or  two  respondents 
at  each  site.  Additionally,  as  the  interviewers  became  conversant  in 
AFC2T2  and  internalized  the  topics  and  probes  of  the  interview  guide,  the 
interviews  became  increasingly  natural,  cogent,  and  productive.  Toward 
the  end  of  the  site  visits  the  interview  guides  actually  become  superfluous 
and,  in  fact,  it  would  have  been  counterproductive  to  depend  on  them  to 
any  large  degree. 

The  observations  were  carried  out  as  planned.  They  were  non -intrusive 
to  ongoing  training  or  exercises.  The  generalized  guide  included  in  this 
appendix  was  suitable  for  all  observation  during  this  study  effort. 

•  Site  Visit  Protocol — The  protocol  to  be  followed  for  each  site  visit  was 
straightforward.  The  steps  were  to  establish  a  point  of  contact,  make  a 
formal  request  and  arrange  the  visit,  conduct  interviews  and  observations, 
and  follow  up  the  visit  with  a  letter  of  appreciation. 

♦ 

As  each  site  was  identified,  a  request  for  cooperation  was  submitted  by 
APHRL.  The  request  asked  that  a  point  of  contact  be  named  for  the  proposed 
visit.  This  individual  was  then  contacted  by  Honeywell  and  the  scope  of  the 
study  was  discussed.  The  primary  purpose  of  the  discussion  was  to  ensure 
that  interviews  with  appropriate  individuals  would  be  arranged  and  that 
observations  of  training  would  be  permitted.  Once  the  details  of  the  visit 
had  been  worked  out,  Honeywell  made  a  written  formal  request  for  on-site 
support,  including  copies  of  a  list  of  general  interview  topics  for  distribution 
to  the  designated  respondents.  This  distribution  preceded  the  actual  visit 
by  one  to  seven  days  in  all  cases. 

During  the  site  visits,  the  point  of  contact  generally  provided  an  orientation 
to  the  unit  or  program.  Interviews  of  knowledgeable  personnel  were 
conducted,  interspersed  with  observations  of  actual  and  training  equipment, 
training,  and  task  performance  of  operationally  ready  teams.  Before 
concluding  the  visit,  the  point  of  contact  and  unit  commander  (occasionally 
these  were  the  same  individual)  were  debriefed  as  to  whether  the  site  visit 
objectives  had  been  met. 
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In  all  cases  during  the  site  visits,  the  support  was  complete  and  enthusiastic. 
The  cooperation  of  all  respondents  and  other  participants  was  commendable. 
A  thank-you  letter  expressing  appreciation  for  support  was  sent  to  each 
point  of  contact  after  the  visit.  The  letter  noted  the  names  of  all  survey 
respondents.  In  several  cases  follow-up  phone  conversations  were  made 
in  order  to  clarify  the  information  obtained. 
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INTERVIEW  GUIDE:  TRAINING  PROGRAM  MANAGER/DEVELOPER 


Respondent  _  Date 

Title 

System  _ 

Organization  _  Phone  _ 

Interviewer (s)  _ _ _ _ _ 


Introduction 

Purpose — Survey  team  training  programs  for  operators  of  systems ; 
identify  strengths  and  determine  issues  requiring  R&D. 

Benefit — Improvement  of  team  training  programs. 


i 


Warm-up 

(How  long  have  you  been  with  this  program?  What  background  did 
you  bring  to  this  system  -  civilian?  military?  What  background 
do  you  have  in  training?  In  the  development  of  C3  systems?) 


» 
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3.  interface — (Does  this  system  interface  with  other  systems? 
Which  systems?) 


4.  position — (Is  this  system  considered  to  be  an  Air  Division, 
Wing,  or  Group  Asset?) 


Personnel 


5. 


Team  Size: 


(How  large  is  the  team  that  operates  this  system? 
How  is  this  team  defined?  Sub-team?  Super  team?) 


6.  Team  composition  and  structure:  (Who  are  the  members  of  the 
team?  What  is  the  composition  of  the  team?  How 
many  officers?  Enlisted?  What  background?  Which 
team  members  report  to  others?) 


Hardware  s,  Functions 


7*  Equipment — (What  computer  is  being  used  and  what  are  its 
specifications?  What  types  of  display  are  used?  What 
equipment  is  used  for  data  input/output?) 


« 


8.  Configuration — (How  many  operator  stations  are  there? 
How  modular  are  the  systems?) 


9.  Function — (How  does  this  equipment  support  the  missions?) 


» 
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Tasks 


10.  Job  description — (We  know  the  team  consists  of  _  peopl 

In  terms  of  the  AFSC  describe  the  job  of  each  of  the 
individuals  who  will  man  the  system?) 


11.  Equipment — (With  what  equipment  does  each  individual  work 
What  are  his  tasks  on  each  piece  of  equipment?) 


12.  Information  reception — (What  sort  of  data, information  or 
direction  will  an  individual  receive?  From  what  source 
will  his  information  come?  In  what  form  will  it  come?) 


13.  Information  processing — (What  is  each  individual  expected  to 
do  with  the  information  he  has  received?  What  decisions  will 
he  make  while  performing  his  tasks?  What  output  will  be 
produce?  In  what  form?  To  whom  will  he  send  this  info?) 


14.  Constraints — (Under  what  sort  of  time  limits  will  he  be 

working?  Under  what  other  constraints?  Which  constraints 
affect  accuracy?  Communication?  How  tolerant  is  the  system 
of  human  error?) 


15.  Team  tasks — (Which  of  each  individuals  responsibilities 
are  carried  out  independent  of  the  other  team  members? 

Which  responsibilities  require  communication  between  team 
members?  Which  responsibilities  require  cooperative 
effort  between  team  members?  Which  responsibilities  would 
you  describe  as  team  tasks?  What  else  should  we  know  about 
each  team  task?) 


Training  Program  Information 
Existing  Training 

16.  Student  background-- (What  lead-in  training  have  students 
received  when  they  enter  training  for  this  program?) 


17.  Training  environment — (What  portion  of  the  existing  training 
is  done  in  a  classroom  setting?  What  does  this  training 
consist  of?  What  portion  of  training  is  done  on  non-operational 
equipment  (simulators/training  devices)?  What  portion  of  the 
training  is  on-the-job  training?  What  does  this  OJT  consist 
of?) 


Training  emphasis — (What  emphasis  is  placed  on  operating 
hardware?  How  much  emphasis  is  placed  on  tasks  done 
independent  of  other  team  members?  How  much  emphasis  on 
system  mission  and  background?  How  much  emphasis  on  tasks 
that  require  communication  between  team  members?  Are 
students  taught  how  their  responsibilities  combine  with 
other  team  members  to  accomplish  the  mission?) 


19.  Training  media  hardware — (How  is  hardware  used  as  part  of 
the  training  system?  How,  if  at  all,  are  simulators  and 
training  devices  used?  Which  tasks  are  trained  using 
simulators?  What  team  skills  are  trained  on  the  device  or 
simulator?  What  tasks  are  trained  using  other  media: 
film,  audio  recordings,  film  strips,  books,  etc.? 

What  are  the  preferred  media?  Which  don't  do  the  job? 

How  easy  is  it  to  modify  the  media?) 


20.  Training  media  software — (What  scenarios  are  used  to  train 
team  skills?  What  other  capabilities  do  these  training 
devices  have — e.g. ,  how  easy  would  it  be  to  change 
scenarios?  What  are  the  software  and  hardware  limitations 
of  the  training  device?  What  sort  of  documentation  is 
required  before  changes  can  be  made?  What  configuration 
control  is  there?) 


21.  Training  strategies  and  objectives — (What  approach  is  used 
to  train  students  in  the  tasks  which  are  done  independent 
of  other  team  members?  How  are  students  taught  the  tasks 
which  require  communication  between  team  members?  What 
other  special  team  training  do  students  receive?  What 
things  can’t  be  taught  at  all?  What  are  the  objectives 
of  training?) 


22.  Course  features — (How  long  is  training?  What  is  the  sequence? 
Is  training  self-paced?  For  individuals?  For  teams? 

How  is  feedback  provided  and  when?  What  feedback  is  provided 
during  hands-on  training  for  individuals?  For  teams?^ 
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23.  Training  of  tasks  information — (Which  tasks  are  taught  only 
partially?  How  and  when  will  the  student  learn  the  full 
task?  Which  tasks  are  taught  in  full?  How  will  the  tasks 
that  are  taught  only  in  part  be  integrated  with  the  tasks 
which  are  taught  in  full?  What  tasks  can  best  be  taught 
in  school?  In  OJT?  Not  at  all  -  only  learned  through 
practice?) 


24.  Course  content — (What  lesson  plans  are  used  in  classroom? 
OJT?  Using  training  devices?  Which  individual  tasks  are 
trained  using  a  specific  plan  of  instruction?  Which  team 
tasks  are  learned  by  the  student  not  through  course  design 
but  as  unplanned  consequences  of  other  training?) 
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Performance  evaluation  &  measures — (What  performance 
measures  are  used?  Are  these  realistic?  How  is 
individual  performance  evaluated?  Team?  Are  norm- 
referenced  or  criterion-based  standards  used  for 
performance  measurement?  What  standards  are  used  to 
evaluate  performances  for  individuals?  What  standards 
are  used  to  evaluate  team  performance?  What  measurements/ 
techniques  are  used  to  evaluate  performance  for 
individuals?  For  teams?) 


Attrition — (What  is  the  attrition  rate  from  this  program? 
When  does  attrition  occur?  What  are  the  primary  causes 
of  attrition?  How  can  attrition  be  lessened?) 


Readiness — (After  training  are  the  students  ready  to  work 
on  operational  equipment  or  do  they  require  further 
training?  How  is  readiness  judged?) 


23.  Training  needs — (What  areas  of  the  training  program  could 

be  improved?  Is  the  syllabus  adequate?  Training  equipment 
sufficient?  Personnel  suited  to  the  job?  Do  you  feel 
there  is  sufficient  emphasis  on  team  training?) 


I 
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Training  Development  and  Implementation 


29.  Organizational  factors — (Which  AF  directives/£rocedures 
are  followed  in  the  development  of  training  objectives? 

What  is  the  management  structure  for  development  of  training 
programs?  How  are  personnel  selected — managers,  instructors, 
students?  What  are  the  procedures  used  to  select,  train, 
and  evalaute  instructors--stan/eval?  Management  personnel?) 


30.  Training  objectives — (How  are  training  objectives  established? 
How  are  the  objectives  established  in  terms  of  student 
performance?  How  are  the  objectives  of  hands-on  training 
determined?) 


» 
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31.  Task  analysis — (Were  task  analyses  done  for  individual 
tasks?  For  team  tasks?  What  methods  are  used  to  do 
team  analyses?  How  is  the  task  analysis  information  used 
in  developing  training  objectives  and  training  requirements? 
When  is  a  task  analysis  done?) 


32.  Skills  identification — (What  methods  were  used  to  identify 
individual  skills  to  be  trained?  To  identify  team 
skills?  How  is  this  information  used?) 


t 


33.  Training  requirements — (What  methods  were  used  to  identify 
individual  training  requirements?  To  identify  team  training 
requirements?  What  makes  it  hard  to  identify  team  training 
requirements?  What  requirements  are  easy  to  identify? 

What  makes  it  hard  to  identify  individual  training  require¬ 
ments?  What  makes  it  easy?  How  were  team  training  require¬ 
ments  used  in  course  development?) 
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Simulator/training  device — (How  is  the  need  for  a  simulator/ 
training  device  determined?  What  methods  are  used  to 
determine  individual  skills  to  be  trained  using  a  simulator? 
Team  skills?  How  and  who  determines  that  a  training 
device  is  the  most  appropriate  method  for  training  certain 
tasks?  How  are  training  device  requirements  defined?  How 
are  scenarios  developed  and  employed?  How  are  training 
devices  or  simulators  used?) 


Hands-on  training — (How  is  the  ratio  of  hands-on  training  to 
classroom  training  established?  How  are  the  objectives 
of  hands-on  training  determes  the  most  effective  training 
approaches?  Who  determines  the  usefulness  of  such  training 
media  as  film  strips,  pamphlets,  fete.?) 


Actual  equipment — (How  is  actual  equipment  used  in  training? 
Is  the  equipment  stimulated?  How?  What  are  the  limitations 
of  actual  equipment?  What  are  the  advantages?) 
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37.  Training  program  implementation-- (What  methods  are  used  to 
take  training  requirements  and  turn  them  into  training 
programs?  What  limits  the  effectiveness  of  this  transition?) 


38.  Team  training — (What  instructional  strategies  are  used  to  train 
team  skills?  What  factors  will  affect  the  efficiency  of 
these  instructional  strategies?  How  are  team  performance 
measures  developed?  What  feedback  techniques  are  used  to 
communicate  the  results  of  team  performance  to  team  members? 

Are  there  any  other  questions  I  should  have  asked?) 


39.  Program  standards — (How  are  program  standards  established? 

How  is  sequencing  established?  Do  the  established 
performance  measures  measure  that  which  needs  to  be  measured? 
Is  there  any  way  to  evaluate  the  appropriateness  of  training 
device  hardware  for  training  the  identified  skills?  of 
training  device  software?) 
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♦  A.  IV* 


M 


Evaluation 

40.  Organizational  factors — (What  AF  procedures  exist  for 

training  evaluation?  When  are  these  procedures  applied? 
What  do  these  procedures  involve?  What  personnel  are 
responsible  for  training  evaluation  and  modification?) 


41.  Information  sources — (What  sources  of  feedback  does  the 
program  have  from  the  field?  From  internal  sources?  How 
is  this  information  used?  Is  there  any  method  available 
to  determine  whether  the  skills  identified  and  being 
trained  for  individuals  are  the  skills  actually  required 
on  the  job?  For  teams?) 
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42.  Appropriateness  of  training — (Is  there  any  attempt  made  to 
determine  whether  the  individual  skills  which  have  been 
identified  and  are  presumably  being  trained  are  in  fact 
the  skills  which  are  being  trained?  What  individual  skills 
may  actually  be  trained?  Is  there  any  method  available 
to  evaluate  how  indiviudal  skill  acquisition  influences 
team  skills  acquisition?) 


43.  Team  performance — (What  performance  measures  are  used  to 
assess  team  skill  acquisition?  On  training  devices?  in 
the  field?  Is  there  any  method  used  to  identify  and 
evaluate  the  team  skills  being  taught  and  to  compare 
them  to  the  skills  identified  as  training  requirements 
in  program  development?  How  is  "readiness"  determined  for 
teams?) 
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44.  Simulator/training  device — (How  is  the  effectiveness  of 
training  devices/simulators  evauated?  Is  software 
evaluated  for  adequacy  in  accomplishing  training?  How 
is  this  done?  When  training  devices  are  used,  how  and 
when  will  the  performance  of  students  on  operational 
equipment  be  evaluated?  Is  there  any  way  to  evaluate  the 
appropriateness  of  training  device  hardware  for  training 
the  identified  skills?  Of  training  device  software?) 


45.  Team  training  device — (When  training  devices  are  used  to 

train  teams — how  and  when  will  the  performance  of  students 
on  operational  equipment  be  evaluated?  Has  there  been  any 
effort  made  to  evaluate  the  effectiveness  of  training  devices 
which  were  designed  to  train  individual  skills  when  these 
devices  are  adapted  for  use  in  training  teams?) 
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46. 


Modification  process — (What  modification  procedures 
exist  for  correcting  device  deficiency?  How  well  do 
these  procedures  work?  What  are  the  procedures  for 
improving  software?  What  other  procedures  exist  to  modify 
training  material?  What  procedures  are  used  to  modify  a 
team  training  program?  What  are  the  difficulties  of 
modifying  a  program?) 


Developing/Modify in g  Training  Program 

This  section  will  be  covered  only  when  a  training  program  is 
in  a  period  of  transition;  either  when  training  is  just 
beginning  to  be  developed  or  when  an  existing  training 
program  is  being  substantialy  modified. 

47.  Where  is  this  system  in  the  acquisition  cycle? 


48.  System  under  modification — (What  training  was  done  before? 
What  modifications  are  being  planned?  Why  are  they  being 
made?  What  impact  will  the  modifications  have  on  training 
objectives  and  requirements?  What  other  impact  is  there?) 
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49. 


Developing  system — (Are  training  considerations  being 
taken  into  account  during  hardware  development?  What 
considerations  are  taken  into  account?  What  sources, 
workbooks ,  AF  procedures/directives  are  being  used  to 
develop  training?) 


» 


50.  Both  systems — (What  strategies  are  being  used  to  develop 
training?  Are  team  training  aspects  being  considered? 
How  will  team  training  be  developed?  When  will  task 
analyses  be  done?  Will  a  team  task  analysis  be  done? 

How  will  performance  measures  be  established?  Training 
device  needs?  Evaluation  procedures?) 


♦ 


Training  Program  Needs 

51.  (In  what  ways  can  this  program  be  improved?  What  information 
would  help  you  in  the  development,  implementation,  and 
evaluation  of  programs?  What  job  aids  have  you  used  before 
and  found  to  be  helpful?  Suppose  we  provided  you  with  .  .  . 


A  B  _  C  _  D 

these  helpful?) 


.  .  would  you  find 


52.  Background — (What  is  your  AFSC/job  description?  What 

does  your  job  actually  require  you  to  do?  How  many  years 
of  experience  do  you  have  in  training  programs?  What 
specialized  vocabulary  have  you  had  to  learn?) 


53.  Job  needs — (Do  you  feel  you  job  could  be  helped  if  you  were 
given  more  information  and  background  in  instructional 
technology?  How  much  time  would  be  allowed  by  your  other 
responsibilities  to  learn  more  about  team  training  and 
instructional  technology?  What  form  of  information  would 
be  most  helpful  to  you — self-study,  seminars,  etc.? 

What  other  time  constraints  do  you  have?  What  sort  of 
information  would  give  you  assistance  in  your  other  job 
needs?) 
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54.  Organizational  constraints — (When  you  are  managing/developing 
a  training  program,  what  sort  of  management  constraints  do 
you  have?  What  about  time  constraints?  How  flexible  is 
this  organization?  What  sort  of  input  do  you  have  in  the 
design/development  process  for  hardware?  How  rapidly 
can  the  organizational  decision  making  process  meet  outside 
demands  or  changing  situations?) 

* 


55.  Decision  points — (What  are  the  critical  decision  points 

when  you  are  developing  a  training  program?  Who  makes  the 
final  decisions  about  training?  Where  do  you  fit  in  the 
decision-making  process?  Who  are  the  key  people  in  this 
process?) 


Validity  Assessment/Interviewer  Comments 


i 
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INTERVIEW  GUIDE  FOR  TRAINING  PROGRAM  MANAGERS/DEVELOPERS 


3 

The  purpose  of  this  survey  is  to  obtain  information  on  existing  C  system  train¬ 
ing  programs  with  special  emphasis  on  team  training  elements  and  programs. 

System  Background  and  Information 
Mission  concept  (stated  mission,  system  interface,  wing  asset?) 

Personnel  (team  size,  definition,  structure) 

Hardware  (equipment,  station  number,  configuration) 

Tasks  (job  descriptions,  equipment  used,  communication  network,  constraints, 
team  tasks) 

Training  Program  Information — Existing  Training 
Student  background  (lead-in  training) 

Training  environment  (classroom  vs.  equipment  vs.  OJT) 

Training  emphasis  (emphasis  on  mission,  team  communication,  individual  tasks) 
Instructional  strategies  (approach  taken  in  training) 

Training  hardware  (use  of  simulators,  other  capabilities,  configuration  control) 
Course  features  (length,  sequence,  feedback  use) 

Course  content  (lesson  plans) 

Performance  evaluation  and  measures  (standards  for  teams,  individuals,  realism) 
Training  needs  (attrition,  syllabus,  training  equipment,  personnel  selection) 
Readiness  (how  judged) 

Training  Program  Development  and  Implementation 
Organizational  factors  (AF  directives,  management  structure,  personnel  selection) 
Training  objectives  (establishment). 

Task  analysis  (who  does  one,  when,  how  used) 

Skills  identification  (methods  used  for  individuals,  teams) 

Training  requirements  (identification  methods) 

Simulator/training  device  need  (how  determined?  scenario  development?) 

Hands-on  training  (how  developed) 

Actual  equipment  (use,  limitations) 

Training  program  implementation  (how,  limitations) 

Team  training  (development) 

Program  standards  (establishment  of  standards,  sequencing,  measures) 
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Training  Program--Eva1uation 
Organizational  factors  (AF  procedures,  personnel) 

Feedback  sources  (field,  internal) 

Appropriateness  of  training  (does  training  teach  identified  skills) 

Team  performance  measures  (readiness,  requirements) 

Simulator  effectiveness  (how  evaluated?  software?  appropriateness  in  training?) 
Team  training  devices  (when  evaluated  on  operational  eauipment) 

Procedures  for  program  modification  (how  hard,  what  is  done) 

Developing/Modifying  Training  Program 
Scope  of  changes 
Purpose  of  changes 
Strategies  for  change 
Respondent  Information 
Background 
Job  needs 

Organizational  constraints 
Decision  points 


INTERVIEW  GUIDE: 


INSTRUCTOR 


Respondent  _  Date 

Title  _ 

Organization  _  Phone 

Interviewer (s)  _ _ 


Introduction 

Purpose — Survey  team  training  programs  for  operators  of 
systems.  Determine  strengths  and  identify  issues  for 
research  and  development. 

Benefit — Potential  increased  support  of  team  training  programs. 


Warm-up 


1.  (How  long  have  you  been  with  this  program?  What  did  you 
do  before  you  came  to  this  program?  How  were  you  assigned? 
Where  did  you  receive  your  instructor  training?  What  are 
your  collateral  duties?  Is  instruction  your  primary  duty?) 
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System  Background  and  Information 


Mission  Concept 

2.  (What  is  the  stated  mission  of  this  system?  What  do  you 
consider  the  most  important  features  of  this  mission: 

How  do  these  features  affect  your  instructing?  Does  this 
system  interface  with  other  systems?  Is  this  system  con¬ 
sidered  to  be  an  Air  Division,  Wing,  or  Group  Asset?) 
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Personnel 


3.  Team  Size:  How  large  is  the  team  that  operates  this  system? 

How  is  this  team  defined?  Sub-team?  Super  team? 


4.  Team  composition  and  structure:  Who  are  the  members  of  the 
team?  What  is  the  composition  of  the  team?  How 
many  officers?  Enlisted?  What  background? 

Which  team  members  report  to  others? 
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Hardware  &  Functions 


* 


5.  (How  does  the  system  hardware  support  the  mission?  What 
types  of  display  are  used?  What  equipment  is  used  for 
data  input/output?  How  many  operator  stations  are  there? 
How  modular  are  the  systems?  What  features  of  the  system 
hardware  make  training  difficult?  From  the  individual 
operator's  point  of  view?  From  a  team  point  of  view?) 
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Tasks 


Individual 

6.  (Job  -  In  tepis  of  AFSC,  would  you  describe  the  job  of  each 
of  the  individuals  who  man  the  system?  How  does  the  job 
support  the  mission  or  various  mission  phases?) 


7.  (Duties  -  What  are  each  team  member's  primary  duties?  What 
are  the  secondary  duties?  How  do  the  duties  vary  as  a 
function  of  mission  phase?  What  is  expected  of  individuals 
in  an  emergency,  e.g. ,  in  the  event  of  the  loss  of  a  team 
member?) 
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(Tasks  -  Discuss  the  tasks  that  enable  performance  of  each 
of  the  duties  described  above.  What  equipment  does  each 
individual  interface  with?  What  is  the  nature  of  the 
tasks  -  procedure  followed?  Decisions  made?  Data/information 
gathered?  processed?  Responses/output  generated?  What  are 
constraints  on  task  speed?  Accuracy?  Based  on  your 
experience,  are  there  any  unique  task  demands?) 


Team 


9.  (What  are  the  coordination  requirements  among  individuals 
in  the  performance  of  their  jobs?  What  specific  duties 
require  coordination/communication  among  team  members?  What 
specific  tasks  require  one  individual  to  provide  data/ 
information  to  another?  To  receive  data/information  from 
another?  what  are  the  constraints  placed  on  inter-individual 
task  performance  -  Accuracy?  Anticipation?  Timing?) 
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Training  Program  Information 
Existing  Training 

(Student  background — What  lead-in  training  have  students 
received  when  they  enter  training  for  this  program? 

What  are  entry  level  skills?  Are  entry  level  skills  the 
right  ones?  Are  entry  skill  levels  api  opriate?) 


Course  Environmen t 


(What  portion  of  the  existing  training  is  done  in  a  class¬ 
room  setting?  What  does  this  training  consist  of?  What 
portion  of  training  is  done  on  non-operational  equipment 
(simulators/training  devices)?  What  are  the  completion 
level  skills  or  qualifications?  How  much  training  is  then 
done  on-the-job?  What  does  OJT  consist  of? 


Coarse  Emphasis 

12.  (What  emphasis  is  placed  on  operating  hardware?  How  much 
emphasis  is  placed  on  tasks  done  independent  of  other  team 
members?  How  much  emphasis  on  system  mission  and  background? 
Hew  much  emphasis  on  tasks  that  require  communication  between 
team  members?  Are  students  taught  how  their  responsibilities 
interface  with  other  team  members?) 


* 
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Training  Media 


13.  (How  is  hardware  used  as  part  of  the  training  system?  How, 
if  at  all,  are  simulators  and  training  devices  used?  Which 
individual  tasks  are  trained  using  simulators?  Team  tasks? 
How  accomplished?  What  capabilities  do  these  training 
devices  have — e.g.,  how  are  scenarios  developed?  How  easy 
would  it  be  to  change  scenarios?  What  are  the  software  and 
hardware  limitations  of  the  training  device?  What  tasks 
are  trained  using  other  media:  film,  audio  recordings, 
film  strips,  books,  etc.?  Which  media  work  best?  Least 
effective?  Which  media  are  used  to  train  team  skills?  Are 
they  satisfactory?  How  easy  or  difficult  are  these  various 
media  to  modify?) 


t 


i 
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Course  Strategies 


(What  approach  is  used  to  train  students  in  the  tasks  which 
are  done  independent  of  other  team  members?  How  are  students 
taught  the  tasks  which  require  communication  between  team 
members?  What  other  special  team  training  do  students  receive?) 


Course  Features 

(How  long  is  training  and  what  sequence  is  used?  How  is 
pacing  achieved  for  individuals?  For  teams?  How  is 
feedback  provided  and  when?  What  feedback  is  provided 
during  hands-on  training  for  individuals?  For  teams? 


Task/Traininq  Info 

16  *  (Which  tasks  are  taught  only  partially?  How  and  when  will 
the  student  learn  the  full  task?  Which  tasks  are  taught  in 
full?  How  will  the  tasks  that  are  taught  only  in  part  be 
integrated  with  the  tasks  which  are  taught  in  full?  Taught 
in  school?  OJT?) 


Course  Content 


17.  (What  lesson  plans  are  used  in  classroom?  How  were  they 
developed?  Who  participated?  What  lesson  plans  exist  for 
using  training  devices?  Which  individual  tasks  are  trained 
using  a  specific  plan  of  instruction?  Which  team  tasks  are 
learned  by  the  student  not  through  course  design  but  as 
unplanned  consequence  of  other  training?  What  can't  be 
trained?) 
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Performance  Evaluation  &  Measures 


18.  (What  performance  measures  are  used?  Are  these  realistic? 

Are  they  appropriate?  How  is  individual  performance 
evaluated?  Team?  What  standards  are  used  to  evaluate 
performance  for  individuals?  Normative?  Criteria-referenced 
What  standards  are  used  to  evaluate  team  performance? 

What  measurements/techniques  are  used  to  evaluate  performance 
for  individuals? for  teams?  After  training  are  the  students 
ready  to  work  on  operational  equipment  or  do  they  require 
further  training?  How  is  readiness  judged?) 
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Instructor  Evaluation 


19.  (How  are  you  evaluated?  Does  this  evaluation  consider  all 
the  relevant  information?  All  of  your  duties?) 


% 


Attrition  f 

20.  (What  is  the  approximate  attrition  rate  from  this  course? 

How  is  attrition  distributed,  i.e.,  when  does  it  occur? 

Why  does  attrition  occur?  What  can  be  done  about  it?) 

% 


* 
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Training  Needs 


21'.  (What  areas  of  the  training  program  could  be  improved?  Are 
resources  and  facilities  sufficient?  Is  the  course  syllabus 
»  doing  the  job?  Do  you  feel  there  is  sufficient  emphasis  on 
team  training?  Are  you  training  what  needs  to  be  trained? 
Are  there  any  questions  I  should  have  asked,  but  didn't?) 


-* 


I 
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Training  Program  Modification 


(When  you  identify  a  training  problem,  how  do  you  go  about 
resolving  it?  What  kinds  of  issues  can  be  resolved 
locally?  Does  the  process  work?  Who  has  the  responsibility 
for  program  modification?  The  authority?) 
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Respondent  Information 

(What  is  your  AFSC/job  description?  What  does  your  job 
actually  require  you  to  do?  What  additional  duties  do  you 
have?  What  priority  does  management  give  to  instructional 
duties?  What  are  your  job  needs?  Is  there  a  specialized 
vocabulary  that  you  use?  How  much  do  you  know  about 
instructional  technology?  Do  you  feel  the  need  to  know 
more?  How  much  time  would  you  be  able  to  find  to  learn 
more  about  team  training  and  instructional  technology?) 
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INTERVIEW  GUIDE  FOR  INSTRUCTOR 


The  purpose  of  this  interview  is  to  obtain  information  on  existing  C  system 
training  programs  with  special  emphasis  on  team  training  elements.  This  in¬ 
formation  can  be  used  to  improve  team  training  programs. 

Respondent  Background 

System  Background  and  Information 

Mission  concept  (stated,  most  important  features— do  these  affect  your 
teaching?) 

Personnel  (definition  of  team,  structure) 

Hardware  and  functions 

Individual  tasks  (AFSC  job  description,  primary  duties,  secondary  duties, 
emergency  situation) 

Description  of  tasks  (equipment  used,  procedures,  decisions  made,  constraints) 
Team  tasks  (coordination/communication  between  individuals) 

Training  Program  Information— Existing  Training 

Student  background  (lead  in  training,  appropriate  skill  levels) 

Course  environment  (classroom  vs.  simulators/training  devices  vs.  OJT) 

Course  emphasis  (mission,  hardware  operation,  conmuni cation) 

Training  media  (use  of  hardware,  simulators,  capabilities,  scenario  develop¬ 
ment,  constraints,  other  media) 

Course  features  (sequence,  pacing,  feedback) 

Task/Training  information  (when  will  integration  take  place) 

Course  content  (lesson  plans,  lesson  plan  development) 

Performance  evaluation  and  measures  (standards  for  individual,  team  readiness) 
Instructor  evaluation  (how  is  instructor  evaluated) 

Attrition  (how  much,  why  what  can  be  done) 

Training  needs  (syllabus,  media,  emphasis  on  team  training) 

Training  Program  Modification 
Procedures  for  modification 
Responsibility,  authority 

Respondent  Information 
AFSC/job  description  vs.  actual  duties 
Time  constraints 
Job  needs 
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INTERVIEW  GUIDE: 


STUDENTS  AND  RECENT  GRADUATES 


Respondents _  Date  _ 

Title  _ _ _ _ 

Commanding  Officer  _ 

System  _ _ _ 

Interviewer (s)  _ _ _ 

Introduction 

3 

Purpose — Survey  team  training  programs  for  operators  of  C 
systems  and  identify  issues  for  R&D. 

Benefit — Improvement  of  training  programs. 


Warm-up 

1.  (How  long  have  you  been  in  the  AF?) 


2.  (What  training  did  you  receive  before  starting  this  course? 
Before  you  joined  the  AF,  where  were  you  in  school?) 
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4.  (Do  you  feel  you  understand  how  this  equipment  which  you're 
learning/have  learned  to  operate  supports  the  mission? 

Do  you  feel  that  it  would  help  you  to  know  that  information?) 


Team  Orientation 

5.  (Do  you  feel  that  you  are  a  member  of  a  team  when  you  are 
operating  this  equipment?  Is  this  something  you  are  taught 
or  does  it  just  happen?) 


6 .  (What  happens  when  you  go  from  training  devices  to  the  real 
equipment?  Do  you  feel  adequately  prepared?  If  not-  why?) 


7.  (Do  you  know  what  the  other  people  you  work  with  are  doing 
and  why  ? ) 
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(Do  you  understand  how  your  job  fits  in  with  the  work  the 
others  are  doing?) 


Evaluation  of  Training 

(Do  you  feel  you  have  a  good  idea  of  how  you're  coming  along 
in  your  course?  Could  you  use  more  feedback?  How  about  as 
a  team?) 


10.  (What  tasks  are  the  easiest  to  learn?  The  most  difficult? 
Which  tasks  are  you  proficient  in?  On  which  tasks  will  you 
require  more  training?  What  type  of  training  will  be 
required?  How  difficult  will  it  be  to  advance  in  skilj. 
level?) 


» 


* 


11.  (Would  you  say  that,  the  job  conditions  are  good?  How  does 
this  affect  your  attitude  about  your  job?) 


4 


12.  (Have  you  been  asked  to  evaluate  the  training  program?  Do 
«  you  feel  that  you  had  any  means  of  changing  the  training 

program?) 
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13.  (What  proportion  of  your  class  didn't  make  it  through 
training?  Why?) 


Graduates 

14.  (Do  you  feel  you  received  adequate  training?  How  difficult 
was  your  adjustment  to  the  real  equipment?) 
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INTERVIEW  GUIDE:  STUDENTS  AND  RECENT  GRADUATES 


Purpose:  Survey  team  training  programs  for  operators  of  systems  and  Identify 
issues  for  R&D. 

*  Benefit:  Improvement  of  training  programs. 


Respondent  Background 

How  long  have  you  been  in  the  AF? 

What  training  did  you  receive  before  starting  this  course?  Before  you  joined 
the  AF,  where  were  you  in  school? 

Mission  Knowledge 

Do  you  know  the  mission  of  the  system  you  are  learning  to  operate?  Would 
you  like  to  know  more? 

Do  you  feel  you  understand  how  this  equipment  which  you're  learning/have 
learned  to  operate  supports  the  mission?  Do  you  feel  that  it  would  help 
you  to  know  that  information? 

Team  Orientation 

Do  you  feel  that  you  are  a  member  of  a  team  when  you  are  operating  this 
equipment?  Is  this  something  you  are  taught  or  does  it  just  happen? 

What  happens  when  you  go  from  training  devices  to  the  real  equipment?  Do 
you  feel  adequately  prepared?  If  not  -  why? 

Do  you  know  what  the  other  people  you  work  with  are  doing  and  why? 

Do  you  understand  how  your  job  fits  in  with  the  work  the  others  are  doing? 

Evaluation  of  Training 

Do  you  feel  you  have  a  good  idea  of  how  you're  coming  along  in  your  course? 
Could  you  use  more  feedback?  How  about  as  a  team? 

What  tasks  are  the  easiest  to  learn?  The  most  difficult?  Which  tasks  are 
you  proficient  in?  On  which  tasks  will  you  require  more  training?  What 
type  of  training  will  be  required?  How  difficult  will  it  be  to  advance 
in  skill  level? 

Would  you  say  that  the  job  conditions  are  good?  How  does  this  affect  your 
attitude  about  your  job? 

Have  you  been  asked  to  evaluate  the  training  program?  Do  you  feel  that  you 
had  any  means  of  changing  the  training  program? 

What  proportion  of  your  class  didn't  make  it  through  training?  Why? 

Graduates 

Do  you  feel  you  received  adequate  training?  How  difficult  was  your  adjust¬ 
ment  to  the  real  equipment? 
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INTERVIEW  GUIDE:  OPERATIONAL  UNIT 

SUPERVISORS  OR  COMMANDERS 

Respondent  _  Date  _ 

Title  _ 

Organization  _  Phone  _ 

Interviewer (s)  _ _ 


Introduction 

Purpose — Survey  team  training  programs  for  operators  of  C3 
systems  and  identify  issues  for  R&D.  Follow-up  with  opera¬ 
tional  units  to  determine  preparadness  of  recent  program 
graduates . 

Benefit — Improvement  of  team  training  programs. 

Warm-up 

1.  (How  long  have  you  been  in  your  present  position?) 

2.  (How  long  have  you  been  associated  with  this  C3  system?) 


Recent  Graduates 


3.  (In  your  opinion  are  the  new  personnel  you  are  receiving 
adequately  trained  for  their  job?  In  what  areas  are  they 
well  prepared?  Less  well  prepared?  How  much  on  the  job 
training  is  necessary  to  bring  new  personnel  up  to  speed?) 


* 


* 


t 


4. 


(How  much  refresher  training  is  carried  out  here? 
skilis/tasks  are  emphasized?) 


What 


♦ 
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(What  are  the  attitudes  of  the  operators  towards  their 
jobs?  How  is  morale?  Do  you  anticipate  any  problems 
retaining  your  good  people?  How  could  retention  be 
improved?  What  are  the  promotion  opportunities?) 


6. 


Training  Improvement 

(How  could  the  current  training  program  be  improved?  Are 
there  any  features  of  your  OJT  program  that  should  be 
incorporated  in  initial  or  school  training?  Do  you  have 
any  input  into  evaluation  or  modification  of  the  training 
program?) 
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(Are  there  any  questions  I  should  have  asked?) 
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INTERVIEW  GUIDE:  OPERATIONAL  UNIT  SUPERVISORS  OR  COMMANDERS 


Purpose:  Survey  team  training  programs  for  operators  of  systems  and  identify 
issues  for  RAD.  Follow-up  with  operational  units  to  determine  preparadness  of 
recent  program  graduates. 

Benefit:  Improvement  of  team  training  pro-rams. 


Respondent  Background 

How  long  have  you  been  in  your  present  position? 

How  long  have  you  been  associated  with  the  system? 

Recent  Graduates 

In  your  opinion,  are  the  new  personnel  you  are  receiving  adequately  trained 
for  their  job?  In  what  areas  are  they  well  prepared?  How  much  on  the  job 
training  is  necessary  to  bring  new  personnel  up  to  speed? 

How  much  refresher  training  is  carried  out  here?  What  skills/tasks  are 
emphasized? 

What  are  the  attitudes  of  the  operators  towards  their  jobs?  How  is  morale? 

Do  you  anticipate  any  problems  retaining  your  good  people?  How  could 
retention  be  improved?  What  are  the  promotion  opportunities? 

Training  Improvement 

How  could  the  current  training  program  be  improved?  Are  there  any  features  of 
your  OJT  program  that  should  be  incorporated  in  initial  or  school  training? 

Do  you  have  any  input  into  evaluation  or  modification  of  the  training  program 

Are  there  any  questions  I  should  have  asked? 


INTERVIEW  GUIDE:  SYSTEMS  PROGRAM  OFFICE  STAFF 
Acquisition  Planners  and  System  Designers 


Respondent  _ _ _  Date  _ 

Title  _ _ . _ — _ 

Organization  _ _ ___________  Phone  _ _ 

Interviewer  (s)  _ _ _ 

Introduction 

Purpose — Survey  team  training  programs  and  future  C3  systems 
to  identify  issues  for  training  R&D. 

Benefit — Increased  readiness  through  anticipation  of  personnel 
and  logistics  needs. 


Warm-up 

1.  (How  long  have  you  been  with  this  program?  What  is  your  back 
ground?  Civilian?  Military?  What  other  experience  have  you 
had  in  the  development  of  C3  systems?) 
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System  Background  and  Information 
Design  Process: 

2.  (How  are  system  requirements/mission/facilities  developed? 
What  role  does  the  user  play  in  the  design  process?  Do 
personnel  or  training  considerations  impact  the  design  of 
the  system?  What  impact  do  training  requirements  have  on 
system  design  and  operators?) 
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Deployment: 

3.  (At  what  sites  is  (will  be)  the  system  deployed?  What  is/ 
will  be  its  echelon  of  command?  where  is  the  system  in  the 
acquisition  cycle?  Are  any  modifications  currently  being 
planned?) 


ft 


Mission  Concept: 

4.  (What  is  the  mission  of  this  system?  What  are  considered 
the  most  important  features  of  the  system?  What  is  the 
system-system  interface?) 


r 


1 
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Personnel : 


5.  (Team  size  -  How  large  is  the  team  that  operates  this  system? 
How  is  the  team  defined?) 


* 


5  (Team  composition  -  Who  are  the  members  of  the  team? 

What  is  the  composition  of  the  team? 
How  many  officers?  Enlisted? 

What  background? 

Which  team  members  report  to  others?) 


» 
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Hardware  and  Functions; 

7.  (What  hardware  is  being  used/developed?  Computers?  Displays? 
Operator  stations?  How  does  it/will  it  support  the  mission? 
Will  any  planned  modifications  affect  the  functions?) 


♦ 


« 


s 


r 


* 
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Tasks 


Individual: 

8.  (Job  -  In  terms  of  the  AFSC,  would  you  describe  the  job  of 
each  of  the  individuals  who  will  man  the  system? 
How  does  the  job  support  the  mission  or  various 
mission  phases?) 


9.  (Duties  -  What  are  the  planned  primary  duties?  What  are 

the  planned  secondary  duties?  How  do  the  duties 
vary  as  a  function  of  mission  phase?  What  is 
expected  of  individuals  in  an  emergency,  e.g., 
in  the  event  of  the  loss  of  a  team  member?) 


10. (Tasks  -  Discuss  the  tasks  that  enable  performance  of  each 

of  the  duties  described  above.  What  equipment  does 
each  individual  interface  with?  What  is  the  nature 
of  the  tasks  -  procedures  followed?  Decisions  made? 
Data/information  gathered,  processed?  Responses/ 
output  generated?  What  are  constraints  on  task 
speed?  Accuracy?  Based  on  your  experience,  are 
there  any  unique  task  demands?  Have  the  tasks/ 
duties  assigned  to  individuals  been  based  on  lessons 
learned  from  your  previous  efforts?  How  do  task 
characteristics  effect  system  design?) 


Team: 


11.  (What  are  the  coordination  requirements  among  individuals 
in  the  performance  of  their  jobs?  What  specific  duties 
require  coordination/communication  among  team  members? 

What  specific  tasks  require  one  individual  to  provide  data/ 
information  to  another?  To  receive  data/information  from 
another?  What  are  the  constraints  placed  on  inter-individual 
task  perforamnce  -  Accuracy?  Anticipation?  Timing?  ) 


* 


< 


t 


« 


12.  (How  tolerant  will  the  system  be  of  human  error?) 
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Respondent  Information 

13.  (What  Is  your  AFSC/job  description?) 


14.  (What  does  your  job  require  you  to  do?) 


15.  (What  are  your  job  needs?) 


16.  (Including  your  current  position,  how  many  years  of  experience 
do  you  have  in  systems  design  or  acquisition?) 


17.  (What  specialized  vocabularies  have  you  had  to  learn  and  use?) 


18.  (How  much  do  you  know  about  instructional  technology?) 


19.  (How  much  time  do  you  feel  you  could  spare  to  learn  more 
about  the  interaction  between  hardware  and  training?  What 
format  would  be  most  useful?  Course?  Reference  material?) 


20.  (What  are  the  critical  decision  points  in  the  development  of 
a  system?) 
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21.  (Who  makes  design  decisions?  Personnel  decisions?  Training? 
Other  decisions?  What  is  the  sign  off  process  for  approval 
of  design?  Personnel?  Is  the  decision  making  process 
flexible  or  rigid?  Is  the  organization  able  to  respond 
rapidly  to  outside  requests?  Is  the  organization  adaptable  when 
confronted  with  new  situations?  What  incentives  are  there 
for  consideration  of  training  issues?  Personnel  issues?  What 
constrains  consideration  of  these  issues  in  system  design 
and  acquisition?  Kow  might  team  training  requirements 
influence  system  design?  Are  there  any  questions  I  should 
have  asked  but  didn't?) 
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OBSERVATION  GUIDE 


System  Representative  _  Date  _ 

Title  _ 

Organization  _  Phone  _ 

Honeywell  Observers _ 

System  Designation  _ 

Equipment/Interface  Characteristics 

Characterization  of  training  device  (e.g.,  simulator, 
actual  equipment,  other) 

1.  Configuration  of  training  device 
a-  Operator  stations 

b.  instructor*  stations 

c.  Processing  system  (e.g. ,  CPU,  software,  peripheral  devices) 


‘Instructor  -  instructor,  referee,  or  controller 
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i  2.  Operator  station  characteristics 

a.  Number  and  modularity  of  operator  stations 

b.  Types  of  information  displayed  to  operator 

i 

c.  Types  of  operator  displays 

d.  Types  of  inputs  operator  can  make 

e.  Types  of  operator  input  devices 

f.  Mechanisms  for  assessing  operator  performance 

g.  Feedback  mechanisms — manual  (e.g.,  verbal),  automatic 

h.  Similarity  of  training  equipment  to  operational 
equipment 

i  Similarity  of  training  functions  to  functions  on 

operational  equipment  in  operational  setting.  Applies 
to  trainer,  simulator,  or  actual  equipment  as  used  for 
training. 

j*  Utilization  of  interactive  CAI  methodologies 

h.  Error  recovery  procedures 
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3.  Instructor  station  characteristics 

a.  Number  and  modularity  of  instructor  stations 

b.  Types  of  information  displayed  to  instructor 

c-  Types  of  instructor  displays 

d.  Types  of  inputs  instructor  can  make 

e.  Types  of  instructor  input  devices 

f.  Methods  by  which  the  instructor  assesses  individual 
and  team  performance 

g.  Methods  by  which  the  instructor  provides  evaluative 
or  diagnostic  feedback  to  individuals  and  teams 

h.  Error  recovery  mechanisms/procedures 

i.  Methods  for  controlling  sequence  of  events  in  scenario 

j.  Methods  for  recording,  retrieving,  displaying  measures 
of  individual  and  team  performance 
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4.  Processing  system  characteristics 


a.  CPU  characteristics  and  functions 

b.  Peripheral  devices  (in  addition  to  operator 
stations)  and  functions 

c.  Software  configuration  and  functions  (e.g., 
of  modular  program  architecture,  language) 


♦ 


i 


4 


1 
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and  instructor 

structured 


Training  Program/Exercise  Characteristics 

5.  Number  of  participants 

6.  Number  of  instructors 

7 .  Duration/Schedule 

8.  Training  objectives 

9.  Performance  measures  (individual  and  group) 

10.  Type  of  training  program  (e.g.,  formal  or  informal  on  the 
job  training,  simulation,  part-task  or  whole-task  exercise) 

11.  Types  of  scenarios  developed  for  the  training  program 
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Operator  Tasks /Constraints 

12.  Number  of  operators  required  to  perform  the  system  mission. 
Minimum  #.  Maximum  #. 

13.  Informal  task  list  for  each  operator  station* 

Note  primary  operations . 

a.  Established  tasks 

b.  Emergent  tasks 

14.  Performance  measures  for  each  task 

15.  Performance  criteria  for  each  task 

16.  Emphasis  placed  on  training  frequent  or  critical  tasks 

17.  Operator  qualifications 


*When  large-scale  exercises  (e.g.,  Blue  Flag,  Red  Flag)  are 
observed,  a  representative  sample  of  operator  stations 
will  be  observed. 
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Intra-Team  Communications 
Types  of  communication 

Communication  channels  (voice,  digital,  visual) 

Information  flow  networks  (Are  there  different  networks 
(teams)  for  different  situations?) 
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Instructor  Activities 

26.  How  does  the  instructor  monitor  individual  and  team 
performance? 

27.  How  does  the  instructor  provide  individual  and  team  feedback? 

28.  How  does  the  instructor  control  the  sequence  of  events? 

29.  How  does  the  instructor  modify  the  level  of  difficulty  of 
the  exercise? 

30.  How  does  the  instructor  control  the  training  hardware  and 
software  systems? 
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Training  Program  Evaluation 


I 


♦ 


t 


# 


» 


31.  How  well  does  the  configuration  of  operator  stations  in 
the  training  device  conform  to  the  stations  in  the 
operational  system? 

32.  Is  the  degree  of  fidelity  of  the  operator  stations  (both 
hardware  and  functions)  appropriate  for  the  training 
objectives? 

33.  Are  individual  and  team  performance  measures  related  to 
training  objectives? 

34.  Can  performance  assessent  be  aided  through  automation? 

35.  Do  operators  receive  immediate  feedback? 

36.  Do  instructor  stations  display  appropriate  information 
and  accept  appropriate  input? 

37.  How  can  the  processing  system  be  improved  to  enhance  the 
instructor's  capabilities  to  monitor  students,  provide 
feedback,  and  control  the  hardware/software  system? 

38.  How  well  does  the  training  device  and  program/exercise 
meet  training  objectives? 

39.  Compare  actual  and  nominal  individual  task  lists. 

40.  Compare  actual  and  nominal  team  task  lists. 

41.  Compare  actual  and  nominal  team  structure  and  information 
flew  networks. 

42.  Does  the  processing  system  have  the  capacity  to  support 
current  training  functions? 

43.  Is  the  processing  system  expandable  to  meet  future  training 
requirements? 

44.  Can  the  generation  of  courseware  for  the  system  be  facilitated 
through  the  use  of  a  standardized  CAI  software  package? 


Validity  Assessment 


APPENDIX  C 


REPRESENTATIVE  COURSE  TRAINING  STANDARDS  (CTS) 


•  f 
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APPENDIX  C 


REPRESENTATIVE  COURSE  TRAINING  STANDARDS  (CTS) 


The  CTS  for  the  ADWC/IWS  APQ  Air  Weapons  Controller  Coarse  and  the 
JSS  Weapons  Controller  Course  are  reproduced  here.  The  proficiency 
level  to  be  achieved  for  each  know  ledge /skill  is  coded  in  the  righthand 
column  of  the  table. 


TABLE  C-l.  CTS:  ADWC/IWS  APQ  AIR  WEAPONS 

CONTROLLER  COURSE  (CTS  ADWC  1741B00) 


t  Scale 

Value 

Proficiency  Level  Definitions 

♦ 

The  instructor  demonstrates  the  task  at  the  D 

terminal  desired  proficiency  level. 

The  student  must  be  prompted  on  how  to  perform  1 
the  task  and  when  to  perform  the  task. 

The  student  is  capable  of  performing  the  task,  2 

but  may  need  prompting  on  when  to  perform  the 
task. 

The  student  is  capable  of  performing  the  task  3 

under  briefed/ standard  situations.  Needs 
prompting  in  nonstandard  situations. 

t  The  student  is  capable  of  performing  the  task  4 

quickly  and  accurately  in  nonstandard  situations. 


Academic  Knowledge  Level  Definitions 

Can  identify  basic  facts  and  terms  about  the  A 

subject. 

Can  explain  the  relationship  of  basic  facts  and  B 

state  general  principles  about  the  subject. 

Can  analyze  facts  and  principles  and  draw  C 

conclusions  about  the  subject. 

Can  evaluate  conditions  and  make  proper  D 

decisions  about  the  subject. 


TABLE  C-l.  CTS:  ADWC/IWS  APQ  AIR  WEAPONS  CONTROLLER 
COURSE  (CTS  ADWC  1741B00)  (continued) 

Academic 

Knowledge 

Level 

Introduction  to  Automated  Control 


1.  Automated  Operational  Units  A 

2.  Flow  of  Information 

a.  Computer  component  functions  a 

b.  Coordination  of  computer  information  B 

3.  Outputs 

a.  Tracking  program  B 

b.  Weapons  program  B 

c.  Voice -data  link  B 


Console  Familiarization 


1.  Displays 

a.  Data  a 

b.  Unique  track  ^ 

c.  Non-track  symbology  a 

d.  Fighter  mission  SID  B 

e.  Offsets  B 

f.  Fighter  mission  TD  B 

2.  Switch  Actions 

a.  Request  information  B 

b.  Induce  change  B 

c.  Initiate  an  action  sequence  B 

3.  WC/WCT  Request 

a.  Request  information  B 

b.  Induce  change  B 

c.  Initiate  an  action  sequence  B 
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TABLE  C-l.  CTS:  ADWC/IWS  APQ  AIR  WEAPONS  CONTROLLER 
COURSE  (CTS  ADWC  1741B00)  (continued) 


* 


* 


f 


% 


* 


Academic 

Knowledge 

Level 

Voice  Control 


1.  When  to  Use  Voice  Control  B 

2.  Airspace  Briefing  C 

3.  Voice  Transmissions  B 

4.  Brevity  Code 

a.  Equipment  B 

b.  Aircraft  mission  situation  B 

c.  Intercept  status  B 

d.  Request  for  information  B 

5.  Computer  Malfunctions  B 

6.  Data  Link/ Radio  Telephone  Procedures  B 

Intercept  Safety  Procedures  (J M  35-200) 

1.  Responsibilities  for  In-Flight  Separation  B 

2.  Minimum  Separation  C 

3.  General  Procedures  B 

4.  Weapons  Controller  Procedures  C 

3.  Fighter  Procedures  C 

6.  Target  Procedures  C 

7.  Special  Procedures  C 

3.  FAA/RAPCON  Separation  Minimums  B 

Air  Traffic  Control  Procedures 

1.  Airspace  Classifications  B 

2.  ATC-ADCF  Radar  Handoffs  C 

3.  Mission  Aircraft  Check-In  C 

4.  Maintaining  Assigned  Airspace  D 

5.  Conflicting  Traffic  D 

6.  ADCF-ATC  Recoveries  C 

Emergency  Procedures 

1.  Recognition  of  an  Emergency  B 

2.  Minimum  Immediate  Required  Actions  B 

3.  Additional  Information/ Actions  B 
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TABLE  C-l.  CTS:  ADWC/IWS  APQ  AIR  WEAPONS  CONTROLLER 
COURSE  (CTS  ADWC  1741B00)  (continued) 


Academic 

Knowledge 

Level 

Fighter  Guidance 


1.  Mission  Description 

a.  Definition  fighter  missions  A 

b.  Definition  computer  phases  A 

c.  Fighter  mission  computer  phases  A 

d.  Fighter  mission  attack  options  B 

2.  Interception  Mission:  Approach  Phase 

a.  Geometry  A 

b.  Profile  selection  B 

c.  Approach  altitude  selection  logic  B 

d.  Profile  monitoring  A 

e.  Approach  parameter  overrides  B 

3.  Interception  Mission:  Attack  Phase 

a.  Entry/ Exit  A 

b.  Cutoff  attack  segment 

1 )  Guidance  B 

2)  Pursuit  phase  B 

3)  Monitoring  B 

c.  Pursuit  attack  option  B 

d.  Stern  attack  option 

1 )  Guidance  B 

2 )  Monitoring  B 

4.  Automatic  Attack  Option  Selection 

a.  Causes  B 

b.  Target  altitude  B 

c.  Attack  option  monitoring  B 

d.  Medium  altitude,  high-speed  logic  B 

e.  Transonic  speed  zone  logic  B 

5.  Attack  Segment  Overrides 

a.  When  the  overrides  may  be  inserted  B 

b.  When  the  overrides  become  effective  B 

c.  Effects  B 

d.  Cancellation  B 


4 


> 


t 
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TABLE  C-l.  CTS:  ADWC/IWS  APQ  AIR  WEAPONS  CONTROLLER 
COURSE  (CTS  ADWC  1741B00)  (continued) 


Academic 

Knowledge 

Level 

Data  Link  (DL)/Fire  Control  System  (FCS) 

1.  Data  Link  Equipment 

a.  Ground  A 

b.  Airborne  A 

2.  Types  of  Data  Link  Information 

a.  Test  A 

b.  Close  control  (CC)  B 

c.  Modified  close  control  (MCC)  A 

3.  Airborne  Data  Link  Checks  A 

4.  Fire  Controi  System 

3.  Controller  Action  Implications  A 

Electronic  Warfare  (EW) 

1.  Radar  Theory  B. 

2.  Passive  Electronic  Warfare  (ELINT)  A 

3.  Active  Electronic  Warfare 

a.  Reradiation  countermeasures  B 

b.  Radiation  countermeasures  B 

4.  Tactical  Countermeasures 

a.  Equipment  B 

b.  Execution  (battle  techniques)  B 

5.  Counter-Countermeasures 

a.  Weapons  Controller  B 

b.  Aircrew  A 

c.  MIJI  Reporting  A 

6.  Passive  Tracking 

a.  Activation  of  passive  tracking  B 

b.  Initiation  B 

c.  Implications  B 

Sage  Orientation 

1.  Location  of  Air  Divisions  A 

2.  Inter- Unit  Relationship  A 

3.  Functional  Areas  within  the  ADCC  A 
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TABLE  C-l.  CTS:  ADWC/IWS  APQ  AIR  WEAPONS  CONTROLLER 
COURSE  (CTS  ADWC  1741B00)  (continued) 


Academic 

Knowledge 

Level 


4.  Weapons  Equipment 

a.  Console  A 

b.  Communications  A 

5.  Use  of  Weapons  Console 

a.  Setup  -4 

b.  Switch  actions  A 

c.  Activations  A 

6.  Displays 

a.  Weapons  SID  A 

b.  Weapons  DID  A 

Special  Missions 

1.  Identification  Intercepts  B 

2.  Coordinated  Attacks  B 


1 .  Planning 

a.  Mission  schedule --Live 

b.  Mission  scheduie--Simu!ation 

c.  Aircrew  Briefing-- Live 

d.  Airspace 

e.  System  limitations 

f.  Weather 

g.  Handoff 

h.  Aircraft  symbology 

i.  Tactics 

j.  Overrides 

k.  Initial  pattern 

l.  Briefed  pattern 

m.  Track  monitor 

n.  Recovery 

o.  Emergencies 

p.  Malfunctions 


Proficiency 

Level 


3 

3 

3 

3 

2 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 
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TABLE  C-l.  CTS:  ADWC/IWS  APQ  AIR  WEAPONS  CONTROLLER 
COURSE  (CTS  ADWC  1741B00)  (concluded) 


Proficiency 

Level 


2.  Pre- Intercept 

a.  Console  setup  3 

b.  WCT/TM  briefing  3 

c.  Initial  symbology /data  association  3 

d.  Handoff  3 

3.  Interception 

a.  Initial  set-up  2 

b.  Cutoff — Heading  2 

c.  Cutoff — Altitude  2 

d.  Cutoff — Speed  2 

e.  Reattack  2 

f.  Stern- -Heading  2 

g.  Stern — Altitude  2 

h.  Stern-'- Speed  2 

i.  CAP--Heading  3 

j.  CAP- -Altitude  3 

k.  CAP- -Speed  3 

L  Overrides  3 

m.  Symbology/ Data  Association- -2  nmi  3 

n.  Symbology/ Data  Association--5  nmi  3 

o.  Safety--Traffic  3 

p.  Safety--Radio  transmissions  (R/T)  3 

q.  Safety- -Spillout  coordination  3 

4.  Post-Intercept 

a.  Pre-Recovery--Planning  3 

b.  Pre- Recovery- -Notification  3 

c.  RTB--Heading  3 

d.  RTB-- Altitude  3 

e.  RTB--Speed  3 

f.  ATC  Recovery — Call  3 

g.  RTB-- Weather /fuel  3 

h.  ATC  clearance  3 

i.  Mission  completion  procedures  3 

5.  Quick  Action  Procedures 

a.  Emergency — Immediate  3 

b.  Emergency — Time  permit  2 

c.  Malfunction- -Ground  equipment  3 
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TABLE  C-2.  CTS  JSS  WEAPONS  CONTROLLER  COURSE 
QUALITATIVE  REQUIREMENTS 


QUALITATIVE  REQUIREMENTS 

CTS  E10SC1744  006 
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TABLE  C-2.  CTS  JSS  WEAPONS  CONTROLLER  COURSE 
QUALITATIVE  REQUIREMENTS  (concluded) 


•  Tasks,  Knowledge,  and  Proficiency  Level 

1.  Introduction/Description 

,  a.  System  overview  B 

b.  ROCC  functional  areas 

1)  Battle  staff  C 

2)  Weapons  C 

3)  Surveillance  C 

4)  Identification  C 

3)  Manual  inputs  C 

6)  Computer  operations  C 

c.  Hardware 

•  1 )  Display  console  B 

2)  Remote  access  terminal  B 

3)  Data  processing  equipment  B 

[  4)  Status /plotting  boards  B 

»  5)  Equipment  configuration  B 

6)  DP  string  utilization  with  complex  B 

2.  Applications  Set 

a.  Use  and  interaction  C 

b.  Terminology  C 

c.  Introduction  of  operational  programs  C 

3.  Consoles 

a.  Physical  characteristics 

>  1 )  Display  C 

2)  Tabular  C 

3 )  Situation  C 

5  4)  Attention  C 

b.  Interaction  of  displays  C 

c.  Operate  display  console  3c 

d.  Analyze  displays  3c 

e.  Switch  Actions  3c 

4.  Crew  Training  2c 
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